Re: wget and WebKit?
On 25.06.22 12:52, Tim Rühsen wrote: On 20.06.22 09:15, Josef Moellers wrote: On 18.06.22 16:11, Tim Rühsen wrote: Moin Josef, Mooi'n Tim, is it possible that 'wget' is linked to something else or is a different binary than expected ? Or maybe 'wget' is a script or alias ? I had asked for straces and they show that "/usr/bin/wget/" is indeed the binary executed. Because a) webkit and wget have nothing in common That was my impression, too. Thanks for confirming! However ... see below ... b) That message you got seems not be originated in GNU Wget Yes, it comes from WebKit (Source/WTF/wtf/posix/ThreadingPOSIX.cpp). I am also discussing this internally with people more familiar with proxy handling. "wget" is linked against libproxy and that may invoke WebKit. So we're now pursuing the "wget" -> "libproxy" -> "WebKit" trail. Does SUSE by any chance add their own patches to wget or any of the linked libraries ? Shame on me: Yes, we do. We have one patch that does indeed add libproxy support. It's been in there for ~12 years :-( The bug is now with the libproxy folks. Sorry for the hassle. Or is there some LD_PRELOAD magic involved ? Signal 10 is SIGUSR1, and wget sets a signal handler for it in src/main.c L122: /* Hangup signal handler. When wget receives SIGHUP or SIGUSR1, it will proceed operation as usual, trying to write into a log file. If that is impossible, the output will be turned off. */ To change wget's behavior, you have to comment out this code: ``` #ifdef SIGUSR1 if (sig == SIGUSR1) signal_name = "SIGUSR1"; #endif ``` and this code at L2094 ``` #ifdef SIGUSR1 signal (SIGUSR1, redirect_output_signal); #endif ``` At least it explains "Overriding existing handler for signal 10. Set JSC_SIGNAL_FOR_GC if you want WebKit to use a different signal" It was one of the first things I suggested, but it didn't help. Josef Maybe switch to SIGUSR2 (signal #12 on x86/ARM) via JSC_SIGNAL_FOR_GC !? Regards, Tim -- SUSE Software Solutions Germany GmbH Frankenstraße 146 90461 Nürnberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg) www.suse.com
Re: wget and WebKit?
On 18.06.22 16:11, Tim Rühsen wrote: Moin Josef, Mooi'n Tim, is it possible that 'wget' is linked to something else or is a different binary than expected ? Or maybe 'wget' is a script or alias ? I had asked for straces and they show that "/usr/bin/wget/" is indeed the binary executed. Because a) webkit and wget have nothing in common That was my impression, too. Thanks for confirming! However ... see below ... b) That message you got seems not be originated in GNU Wget Yes, it comes from WebKit (Source/WTF/wtf/posix/ThreadingPOSIX.cpp). I am also discussing this internally with people more familiar with proxy handling. "wget" is linked against libproxy and that may invoke WebKit. So we're now pursuing the "wget" -> "libproxy" -> "WebKit" trail. Thanks, Josef -- SUSE Software Solutions Germany GmbH Frankenstraße 146 90461 Nürnberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg) www.suse.com
wget and WebKit?
Mooi'n, I am trying to make some sense out of a bug where wget crashes consistently when run as an unprivileged user and does not crash when run as root. When wget is run, there is this message: Overriding existing handler for signal 10. Set JSC_SIGNAL_FOR_GC if you want WebKit to use a different signal Has anyone seen this before and/or has experience with WebKit and wget? If only I would understand what WebKit's role might be here ... Thanks, Josef -- SUSE Software Solutions Germany GmbH Frankenstraße 146 90461 Nürnberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg) www.suse.com
Re: CVE-2021-31879
On 04.05.21 08:59, Josef Moellers wrote: > Hi, > > I'm currently trying to tackle the CVE about passing credentials to > redirected servers. > I wonder if it may be necessary to be able to disable this feature, if > one trusts the servers, ie if some kind of command-line option might be > necessary. After having run up and down the wrong alley for a few days (I had been thinking that these were the "real" credentials, eg passed with "https://user:pass@host/;), I have finally found a solution: 1) initializing "location_changed" to 0 in src/retr.c::retrieve_url() 2) passing the current value of "location_changed" to src/http.c::http_loop() 3) passing it on to gethttp() 4) preventing setting up any dangerous user header lines (eg "Authorization:", "Cookie:") when "location_changed" is non-0. An alternative could be to just set up every header as is done now and THEN discard anything dangerous, ie after adding the user headers go through req->headers[] and throw away any header with name "Authorization" or "Cookie". The question remains is if this should be done unconditionally or whether it should be made configurable, eg through a "--trust-redirections" option. Thanks, Josef -- SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
CVE-2021-31879
Hi, I'm currently trying to tackle the CVE about passing credentials to redirected servers. I wonder if it may be necessary to be able to disable this feature, if one trusts the servers, ie if some kind of command-line option might be necessary. Josef -- SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
Re: [Bug-wget] [Secunia Research] GNU wget Vulnerability Report - Request for Details
On 04.04.19 09:27, Tim Rühsen wrote: > On 4/4/19 3:14 AM, Secunia Research wrote: >> Hello, >> >> We are currently processing a report published by a third-party [1] for GNU >> wget and are currently evaluating it to publish a Secunia Advisory for this. >> Please see the original report for details. >> >> We would appreciate to receive your comments on those issues before we >> publish our advisory based on this information. >> >> * Can you confirm the vulnerability? > > Yes Can you please elaborate what EXACTLY the vulnerability is? I have searched through the (quite hefty) diff between 1.20.1 and 1.20.2 and have found only 4 differences that may be viewed as these, but the changes in src/ftp-ls.c and src/http.c do not fix a vulnerability. The CVE-entry is not quite helpful, to say the least ;-) Thanks, Josef -- SUSE Linux GmbH Maxfeldstrasse 5 90409 Nuernberg Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) signature.asc Description: OpenPGP digital signature
Re: [Bug-wget] Read error (Success)?
On 20.11.18 16:29, Dale R. Worley wrote: > "Tony Lewis" writes: >> I'm getting the following error and don't understand what it's trying to >> tell me: >> >> Read error at byte 97430 (Success) >> >> What could the server be doing to cause wget to report an error with the >> details being "Success"? > > My guess is that the server's response starts with a message including > the description "Success". No, it usually comes from a situation where some error is detected and the string representing the value of "errno" is output, but "errno" does not reflect the error and is 0 -> "Success". Try printing "strerror(0)": "Success". Josef
[Bug-wget] Wget not ignoring the return value of a void function in 1.19.5
Hi,
While trying to upgrade to 1.19.5, we found a bug in wget (src/host.c)
where the (non-existing) return value of a void function is assigned to
a variable.
A patch is appended.
Josef
Index: wget-1.19.5/src/host.c
===
--- wget-1.19.5.orig/src/host.c
+++ wget-1.19.5/src/host.c
@@ -732,7 +732,10 @@ wait_ares (ares_channel channel)
ares_process (channel, _fds, _fds);
}
if (timer)
-timer = ptimer_destroy (timer);
+ {
+ptimer_destroy (timer);
+timer = NULL;
+ }
}
static void
Re: [Bug-wget] looking for information
On 18.12.2017 04:17, Iurie wrote: > hello, > i have not found a place like general mailing list for wget nor a forum, > but my problem is this: > > is that possible to give a .txt file with web links as a input to the wget > and it do download all content from the links from the .txt file (from each > link from the .txt file to download everything that that link has)? to > download that content to a specific folder in computer. Have a look at the "-i file" option. Josef
Re: [Bug-wget] Problems building wget2
On 27.09.2017 14:45, Tim Rühsen wrote: > Hi Josef, > > please see README.md on how to build wget2. Thanks! Josef > > If you installed from git repo (https://gitlab.com/gnuwget/wget2.git) then > > ./bootstrap > > ./configure > > make > > make check > > > With Best Regards, Tim > > > > On 09/27/2017 02:41 PM, Josef Moellers wrote: >> I was trying to see if a bug I fixed in wget also happens to be in >> wget2, but I have problems building wget2! >> >> When I call "autoconf" to build the configure script, I get these error >> messages: >> >> configure.ac:12: error: possibly undefined macro: AM_INIT_AUTOMAKE >> configure.ac:104: error: possibly undefined macro: AM_PROG_CC_C_O >> configure.ac:267: error: possibly undefined macro: AS_IF >> configure.ac:291: error: possibly undefined macro: AC_DEFINE >> configure.ac:300: error: possibly undefined macro: AM_CONDITIONAL >> configure.ac:398: error: possibly undefined macro: AC_SEARCH_LIBS >> configure.ac:400: error: possibly undefined macro: AC_MSG_WARN >> >> I have tried on a SLES12-SP3 and a Kubuntu 16.04 >> >> Josef >> >> >
[Bug-wget] Problems building wget2
I was trying to see if a bug I fixed in wget also happens to be in wget2, but I have problems building wget2! When I call "autoconf" to build the configure script, I get these error messages: configure.ac:12: error: possibly undefined macro: AM_INIT_AUTOMAKE configure.ac:104: error: possibly undefined macro: AM_PROG_CC_C_O configure.ac:267: error: possibly undefined macro: AS_IF configure.ac:291: error: possibly undefined macro: AC_DEFINE configure.ac:300: error: possibly undefined macro: AM_CONDITIONAL configure.ac:398: error: possibly undefined macro: AC_SEARCH_LIBS configure.ac:400: error: possibly undefined macro: AC_MSG_WARN I have tried on a SLES12-SP3 and a Kubuntu 16.04 Josef
Re: [Bug-wget] http server responding with 416 but file was not transferred completely
On 18.09.2017 16:48, Tim Rühsen wrote: > Hi Josef, > > pushed your patch - thanks for your contribution. Welcome > BTW, the example from your original post doesn't show that wrong server > behavior any more. Maybe the server got fixed meanwhile !? I guess so. Probably a glitch. That's why I posted the little server that couldn't ;-) Josef > On 09/14/2017 07:47 PM, Josef Moellers wrote: >> On 14.09.2017 17:06, Tim Rühsen wrote: >>> On 09/14/2017 12:11 PM, Josef Moellers wrote: >>>> On 14.09.2017 10:12, Tim Rühsen wrote: >>>>> On 09/14/2017 09:53 AM, Josef Moellers wrote: >>>>>> Hi, >>>>>> >>>>>> We have seen (at least) one server who has >>>>>> Accept-Ranges: bytes >>>>>> in his header but, upon receiving a request with >>>>>> Range: bytes=23068672- >>>>>> responds with >>>>>> HTTP/1.1 416 Requested Range Not Satisfiable >>>>>> although the file was not transferred completely! >>>>>> >>>>>> wget then claims that >>>>>> The file is already fully retrieved; nothing to do. >>>>>> >>>>>> E.g. >>>>>> run >>>>>> wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >>>>>> the, after a couple of MB, abort the transfer and then continue the >>>>>> download: >>>>>> wget --continue >>>>>> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >>>>>> >>>>>> Maybe the check in src/http.c: >>>>>> 3821 if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE >>>>>> 3822 || (!opt.timestamping && hs->restval > 0 && statcode == >>>>>> HTTP_STATUS_OK >>>>>> 3823 && contrange == 0 && contlen >= 0 && hs->restval >= >>>>>> contlen)) >>>>>> >>>>>> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an >>>>>> incomplete file should show something like >>>>>> >>>>>> "download continue failed, file incomplete" >>>>> >>>>> Well, that would be ok for this special server. >>>>> >>>>> Normally 416 together with a server timestamp matching the file's >>>>> timestamp means that the file is complete (as far as the client can >>>>> judge from HTTP). >>>>> >>>>> IMO, if the server is broken (or misbehaves) then the server should be >>>>> repaired. Except it is a very common misbehavior. In which case we could >>>>> consider a work-around on the client side. >>>>> >>>> >>>> So I humbly propose the attached patch. >>>> I tried to create a pull request, but got a 403. >>> >>> Thanks for the patch - I'll test it in the next days. >> >> I have attached a simple webserver that simulates the error: >> when a request with a Range comes in, it replies with 416 and also >> returns an unsanely huge Content-Length. You'll need glib2 and >> microhttpd for it to build. >> >> I was able to reproduce the issue with this server and check that the >> patch fixes it by causing wget to retry (until --retries is exhausted). >> >>> BTW, we currently work on Wget2 where we have a related issue, if you >>> like to take a look at it: https://gitlab.com/gnuwget/wget2/issues/278 >> >> I'll do that. >> >> Josef >> >
Re: [Bug-wget] http server responding with 416 but file was not transferred completely
On 15.09.2017 09:34, Josef Moellers wrote: > On 14.09.2017 17:06, Tim Rühsen wrote: >> On 09/14/2017 12:11 PM, Josef Moellers wrote: >>> On 14.09.2017 10:12, Tim Rühsen wrote: >>>> On 09/14/2017 09:53 AM, Josef Moellers wrote: >>>>> Hi, >>>>> >>>>> We have seen (at least) one server who has >>>>> Accept-Ranges: bytes >>>>> in his header but, upon receiving a request with >>>>> Range: bytes=23068672- >>>>> responds with >>>>> HTTP/1.1 416 Requested Range Not Satisfiable >>>>> although the file was not transferred completely! >>>>> >>>>> wget then claims that >>>>> The file is already fully retrieved; nothing to do. >>>>> >>>>> E.g. >>>>> run >>>>> wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >>>>> the, after a couple of MB, abort the transfer and then continue the >>>>> download: >>>>> wget --continue >>>>> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >>>>> >>>>> Maybe the check in src/http.c: >>>>> 3821 if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE >>>>> 3822 || (!opt.timestamping && hs->restval > 0 && statcode == >>>>> HTTP_STATUS_OK >>>>> 3823 && contrange == 0 && contlen >= 0 && hs->restval >= >>>>> contlen)) >>>>> >>>>> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an >>>>> incomplete file should show something like >>>>> >>>>> "download continue failed, file incomplete" >>>> >>>> Well, that would be ok for this special server. >>>> >>>> Normally 416 together with a server timestamp matching the file's >>>> timestamp means that the file is complete (as far as the client can >>>> judge from HTTP). >>>> >>>> IMO, if the server is broken (or misbehaves) then the server should be >>>> repaired. Except it is a very common misbehavior. In which case we could >>>> consider a work-around on the client side. >>>> >>> >>> So I humbly propose the attached patch. >>> I tried to create a pull request, but got a 403. >> >> Thanks for the patch - I'll test it in the next days. > > Hold the press ... > While the fix catches some mis-behaviour of the remote server, we have > another issue: > when "-N" is specified, "--continue" apparently does not contine when > the file's modification date has not changed. Maybe deciding that "timestamping" and "always_rest" are mutually exclusive or disabling "timestamping" if "always_rest" is set? Josef signature.asc Description: OpenPGP digital signature
Re: [Bug-wget] http server responding with 416 but file was not transferred completely
On 14.09.2017 17:06, Tim Rühsen wrote: > On 09/14/2017 12:11 PM, Josef Moellers wrote: >> On 14.09.2017 10:12, Tim Rühsen wrote: >>> On 09/14/2017 09:53 AM, Josef Moellers wrote: >>>> Hi, >>>> >>>> We have seen (at least) one server who has >>>> Accept-Ranges: bytes >>>> in his header but, upon receiving a request with >>>> Range: bytes=23068672- >>>> responds with >>>> HTTP/1.1 416 Requested Range Not Satisfiable >>>> although the file was not transferred completely! >>>> >>>> wget then claims that >>>> The file is already fully retrieved; nothing to do. >>>> >>>> E.g. >>>> run >>>> wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >>>> the, after a couple of MB, abort the transfer and then continue the >>>> download: >>>> wget --continue >>>> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >>>> >>>> Maybe the check in src/http.c: >>>> 3821 if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE >>>> 3822 || (!opt.timestamping && hs->restval > 0 && statcode == >>>> HTTP_STATUS_OK >>>> 3823 && contrange == 0 && contlen >= 0 && hs->restval >= >>>> contlen)) >>>> >>>> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an >>>> incomplete file should show something like >>>> >>>> "download continue failed, file incomplete" >>> >>> Well, that would be ok for this special server. >>> >>> Normally 416 together with a server timestamp matching the file's >>> timestamp means that the file is complete (as far as the client can >>> judge from HTTP). >>> >>> IMO, if the server is broken (or misbehaves) then the server should be >>> repaired. Except it is a very common misbehavior. In which case we could >>> consider a work-around on the client side. >>> >> >> So I humbly propose the attached patch. >> I tried to create a pull request, but got a 403. > > Thanks for the patch - I'll test it in the next days. Hold the press ... While the fix catches some mis-behaviour of the remote server, we have another issue: when "-N" is specified, "--continue" apparently does not contine when the file's modification date has not changed. I'll investigate this, but if anyone has an idea ;-) Josef signature.asc Description: OpenPGP digital signature
Re: [Bug-wget] http server responding with 416 but file was not transferred completely
On 14.09.2017 17:06, Tim Rühsen wrote: > On 09/14/2017 12:11 PM, Josef Moellers wrote: >> On 14.09.2017 10:12, Tim Rühsen wrote: >>> On 09/14/2017 09:53 AM, Josef Moellers wrote: >>>> Hi, >>>> >>>> We have seen (at least) one server who has >>>> Accept-Ranges: bytes >>>> in his header but, upon receiving a request with >>>> Range: bytes=23068672- >>>> responds with >>>> HTTP/1.1 416 Requested Range Not Satisfiable >>>> although the file was not transferred completely! >>>> >>>> wget then claims that >>>> The file is already fully retrieved; nothing to do. >>>> >>>> E.g. >>>> run >>>> wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >>>> the, after a couple of MB, abort the transfer and then continue the >>>> download: >>>> wget --continue >>>> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >>>> >>>> Maybe the check in src/http.c: >>>> 3821 if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE >>>> 3822 || (!opt.timestamping && hs->restval > 0 && statcode == >>>> HTTP_STATUS_OK >>>> 3823 && contrange == 0 && contlen >= 0 && hs->restval >= >>>> contlen)) >>>> >>>> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an >>>> incomplete file should show something like >>>> >>>> "download continue failed, file incomplete" >>> >>> Well, that would be ok for this special server. >>> >>> Normally 416 together with a server timestamp matching the file's >>> timestamp means that the file is complete (as far as the client can >>> judge from HTTP). >>> >>> IMO, if the server is broken (or misbehaves) then the server should be >>> repaired. Except it is a very common misbehavior. In which case we could >>> consider a work-around on the client side. >>> >> >> So I humbly propose the attached patch. >> I tried to create a pull request, but got a 403. > > Thanks for the patch - I'll test it in the next days. I have attached a simple webserver that simulates the error: when a request with a Range comes in, it replies with 416 and also returns an unsanely huge Content-Length. You'll need glib2 and microhttpd for it to build. I was able to reproduce the issue with this server and check that the patch fixes it by causing wget to retry (until --retries is exhausted). > BTW, we currently work on Wget2 where we have a related issue, if you > like to take a look at it: https://gitlab.com/gnuwget/wget2/issues/278 I'll do that. Josef 416-webserver.tgz Description: application/compressed-tar signature.asc Description: OpenPGP digital signature
Re: [Bug-wget] http server responding with 416 but file was not transferred completely
On 14.09.2017 10:12, Tim Rühsen wrote:
> On 09/14/2017 09:53 AM, Josef Moellers wrote:
>> Hi,
>>
>> We have seen (at least) one server who has
>> Accept-Ranges: bytes
>> in his header but, upon receiving a request with
>> Range: bytes=23068672-
>> responds with
>> HTTP/1.1 416 Requested Range Not Satisfiable
>> although the file was not transferred completely!
>>
>> wget then claims that
>> The file is already fully retrieved; nothing to do.
>>
>> E.g.
>> run
>> wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO
>> the, after a couple of MB, abort the transfer and then continue the
>> download:
>> wget --continue
>> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO
>>
>> Maybe the check in src/http.c:
>> 3821 if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE
>> 3822 || (!opt.timestamping && hs->restval > 0 && statcode ==
>> HTTP_STATUS_OK
>> 3823 && contrange == 0 && contlen >= 0 && hs->restval >= contlen))
>>
>> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an
>> incomplete file should show something like
>>
>> "download continue failed, file incomplete"
>
> Well, that would be ok for this special server.
>
> Normally 416 together with a server timestamp matching the file's
> timestamp means that the file is complete (as far as the client can
> judge from HTTP).
>
> IMO, if the server is broken (or misbehaves) then the server should be
> repaired. Except it is a very common misbehavior. In which case we could
> consider a work-around on the client side.
>
So I humbly propose the attached patch.
I tried to create a pull request, but got a 403.
Josef
Index: wget-1.19.1/src/http.c
===
--- wget-1.19.1.orig/src/http.c
+++ wget-1.19.1/src/http.c
@@ -3819,6 +3819,16 @@ gethttp (const struct url *u, struct url
}
if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE
+ && hs->restval < (contlen + contrange))
+{
+ /* The file was not completely downloaded,
+ yet the server claims the range is invalid.
+ Bail out. */
+ CLOSE_INVALIDATE (sock);
+ retval = RANGEERR;
+ goto cleanup;
+}
+ if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE
|| (!opt.timestamping && hs->restval > 0 && statcode == HTTP_STATUS_OK
&& contrange == 0 && contlen >= 0 && hs->restval >= contlen))
{
Re: [Bug-wget] http server responding with 416 but file was not transferred completely
On 14.09.2017 10:12, Tim Rühsen wrote: > On 09/14/2017 09:53 AM, Josef Moellers wrote: >> Hi, >> >> We have seen (at least) one server who has >> Accept-Ranges: bytes >> in his header but, upon receiving a request with >> Range: bytes=23068672- >> responds with >> HTTP/1.1 416 Requested Range Not Satisfiable >> although the file was not transferred completely! >> >> wget then claims that >> The file is already fully retrieved; nothing to do. >> >> E.g. >> run >> wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >> the, after a couple of MB, abort the transfer and then continue the >> download: >> wget --continue >> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO >> >> Maybe the check in src/http.c: >> 3821 if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE >> 3822 || (!opt.timestamping && hs->restval > 0 && statcode == >> HTTP_STATUS_OK >> 3823 && contrange == 0 && contlen >= 0 && hs->restval >= contlen)) >> >> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an >> incomplete file should show something like >> >> "download continue failed, file incomplete" > > Well, that would be ok for this special server. > > Normally 416 together with a server timestamp matching the file's > timestamp means that the file is complete (as far as the client can > judge from HTTP). > > IMO, if the server is broken (or misbehaves) then the server should be > repaired. Except it is a very common misbehavior. In which case we could > consider a work-around on the client side. OK, so I'll have a go at it. Looks simple enough (famous last words ;-) ) Josef
Re: [Bug-wget] What are the tests testing?
On 12.06.2017 10:00, Tim Rühsen wrote:
> Hi Josef,
>
>
> On 06/12/2017 09:23 AM, Josef Moellers wrote:
>> Hello Tim,
>>
>> Thanks for the reply.
>>
>> On 10.06.2017 13:36, Tim Rühsen wrote:
>>> On Freitag, 9. Juni 2017 17:02:15 CEST Josef Moellers wrote:
>>>> Hi,
>>>
>>> Hi Josef,
>>>
>>>> I'm currently trying to build test suites for openQA.
>>>> One of the candidates is wget and, luckily, it already provides quite an
>>>> extensive test suite.
>>>> I have successfully built an RPM which has all that is needed for the
>>>> tests.
>>>> One test, Test-ftp-iri-fallback.px, fails on SLES12-SP2 and I can't see
>>>> why.
>>>
>>> Look at tests/Test-ftp-iri-fallback.log, if you can't interpret the content
>>> send it here.
>>
>> I cannot find any such file, no *.log" anywhere in the vicinity of the
>> tests.
>
> Ok, the .log files just contain the output of each single test when
> tested with 'make check'. If you use run-px, copy & pasting from the
> console is the right thing to do.
>
>> Ah ... maybe I should have addede that I'm working on a slightly older
>> version of wget: 1.14, which we ship with SLES12.
>
> So I compiled 1.14 (git tag v1.14) and used run-px to run the test suite
> - but still can't reproduce the problem (Debian unstable here, `locale`
> shows all set to 'en_US.UTF-8').
>
>>
>> 227 Entering Passive Mode (127,0,0,1,155,189)
>> --> RETR français.txt^M
>>
>> 150 Opening ASCII mode data connection.
>> Length: 12 (unauthoritative)
>>
>> 0K 100% 2.95M=0s
>>
>> 226 File retrieval complete. Data connection has been closed.
>> 2017-06-09 16:42:53 (2.95 MB/s) - â<80><98>français.txtâ<80><99> saved [12]
>>
>> Test failed: file français.txt not downloaded
>
> My out put looks identical except the these last lines:
>
> 227 Entering Passive Mode (127,0,0,1,175,123)
> --> RETR français.txt
>
> 150 Opening ASCII mode data connection.
> Length: 12 (unauthoritative)
>
> 0K 100% 2.05M=0s
>
> 226 File retrieval complete. Data connection has been closed.
> 2017-06-12 09:39:27 (2.05 MB/s) - âfran\347ais.txtâ saved [12]
>
> Test successful.
>
>
> I just can guess:
> - something with your locale (what does the 'locale' command output ?)
LANG=POSIX
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
Changing everything to en_US.UTF-8 doesn't help:
I added "export LC...=en_US.UTF-8" lines prior to running the test.
I must admit, I never really understood this "locale" thingy, it bites
my whenever I come close.
> - something with iconv() function
>
> Does the same test fail if you use Wget 1.19.1 ?
I can't get it to build on SLES12 as it does not have libidn2(-devel)!
I'll keep on trying, but until then ...
> And from your 'update 2':
>> If I add the option "-O fran${ccedilla_l1}ais.txt" to the cmdline,
>> then the test succeeds:
>
> Of course it does ;-) You simply created the expected output file...
> (circumventing the real test.)
:-(
Josef
signature.asc
Description: OpenPGP digital signature
Re: [Bug-wget] What are the tests testing?
On 12.06.2017 09:37, Josef Moellers wrote:
> On 12.06.2017 09:23, Josef Moellers wrote:
>> Hello Tim,
>>
>> Thanks for the reply.
>
> I ran just this test under strace:
> top_srcdir=/var/opt/wget-tests
> test=Test-ftp-iri-fallback.px
> strace -fo test.out perl -I$top_srcdir/tests $top_srcdir/tests/$test
> $top_srcdir
>
> and it seems that it never tests for fran\347ais.txt, but it does test
> for fran\303\247ais.txt:
> 3511 lstat("fran\303\247ais.txt", {st_mode=S_IFREG|0644, st_size=12,
> ...}) = 0
> 3511 geteuid() = 0
> 3511 lstat("fran\303\247ais.txt", {st_mode=S_IFREG|0644, st_size=12,
> ...}) = 0
> 3511 unlink("fran\303\247ais.txt") = 0
>
> So, it does fall back to the UTF8 version of the filename but still
> checks for the iso-8859-1 filename!
Update^2:
If I add the option "-O fran${ccedilla_l1}ais.txt" to the cmdline, then
the test succeeds:
:
226 File retrieval complete. Data connection has been closed.
2017-06-12 09:38:43 (3.07 MB/s) - ‘fran\347ais.txt’ saved [12]
Test successful.
-end of output-
Josef
>> On 10.06.2017 13:36, Tim Rühsen wrote:
>>> On Freitag, 9. Juni 2017 17:02:15 CEST Josef Moellers wrote:
>>>> Hi,
>>>
>>> Hi Josef,
>>>
>>>> I'm currently trying to build test suites for openQA.
>>>> One of the candidates is wget and, luckily, it already provides quite an
>>>> extensive test suite.
>>>> I have successfully built an RPM which has all that is needed for the
>>>> tests.
>>>> One test, Test-ftp-iri-fallback.px, fails on SLES12-SP2 and I can't see
>>>> why.
>>>
>>> Look at tests/Test-ftp-iri-fallback.log, if you can't interpret the content
>>> send it here.
>>
>> I cannot find any such file, no *.log" anywhere in the vicinity of the
>> tests.
>>
>> Ah ... maybe I should have addede that I'm working on a slightly older
>> version of wget: 1.14, which we ship with SLES12.
>>
>> NB I run the tests by calling
>> run-px /var/opt/wget-tests
>> The tests are installed in /var/opt/wget-tests/tests and the wget binary
>> is in /var/opt/wget-tests/src (although I would have preferred to use
>> the system's own wget, but that's a thing to be considered later).
>>
>> I want to run just the tests in an openQA environment to aid in
>> integration testing. To that end, I am building an RPM with just enough
>> to run the tests:
>> tests/run-px
>> tests/unit-tests
>> tests/Test-*
>> tests/*.pm
>> tests/Makefile (not used)
>> tests/WgetFeature.cfg
>> tests/WgetTest.pm.in
>> tests/certs/*
>> src/wget
>>
>>>> Is there a list describing exactly what each test checks and what a
>>>> failure means?
>>>
>>> Each test should self-contain a short description of it's purpose,
>>> sometimes
>>> these are missing (accidentally).
>>
>> The accident must have happened here ;-)
>> NB The only difference I find between the 1.14 and the 1.19 versions of
>> this test is that the 1.19 version has the "name" hash tag in the
>> "FTPTest->new()" call.
>>
>>> Test-ftp-iri-fallback tries to FTP-download a file containing non-ASCII
>>> char(s).
>>> The behavior of Wget (with IRI support) is to convert the file name to
>>> UTF-8
>>> for using with a RETR command.
>>> This should fail with a "550 file not found".
>>
>> It does.
>>
>>> Now Wget falls back to the unconverted file name and tries RETR again -
>>> this
>>> should succeed (we told the FTP test server to know this file name).
>>
>> This indeed succeeds, but in the end:
>>
>> Test failed: file français.txt not downloaded
>>
>>
>> As I cannot find any log file, here's the output the test produced
>> (using cut-and-past from the ssh tty):
>>
>> Running Test-ftp-iri-fallback.px
>>
>> Running test Test-ftp-iri-fallback
>> Calling ../src/wget --local-encoding=iso-8859-1 -S
>> ftp://localhost:39938/français.txt
>> --2017-06-09 16:42:53-- ftp://localhost:39938/fran%C3%A7ais.txt
>>=> â<80><98>français.txtâ<80><99>
>> Resolving localhost (localhost)... ::1, 127.0.0.1
>> Connecting to localhost (localhost)|::1|:39938... failed: Connection
>> refused.
>> Connecting to localhost (localhost)|127.0.0.1|:39938... connected.
>> Logging in as anonymous ...
>> 220 GNU Wget Testing FTP Server ready.
>>
Re: [Bug-wget] What are the tests testing?
On 12.06.2017 09:23, Josef Moellers wrote:
> Hello Tim,
>
> Thanks for the reply.
I ran just this test under strace:
top_srcdir=/var/opt/wget-tests
test=Test-ftp-iri-fallback.px
strace -fo test.out perl -I$top_srcdir/tests $top_srcdir/tests/$test
$top_srcdir
and it seems that it never tests for fran\347ais.txt, but it does test
for fran\303\247ais.txt:
3511 lstat("fran\303\247ais.txt", {st_mode=S_IFREG|0644, st_size=12,
...}) = 0
3511 geteuid() = 0
3511 lstat("fran\303\247ais.txt", {st_mode=S_IFREG|0644, st_size=12,
...}) = 0
3511 unlink("fran\303\247ais.txt") = 0
So, it does fall back to the UTF8 version of the filename but still
checks for the iso-8859-1 filename!
Josef
> On 10.06.2017 13:36, Tim Rühsen wrote:
>> On Freitag, 9. Juni 2017 17:02:15 CEST Josef Moellers wrote:
>>> Hi,
>>
>> Hi Josef,
>>
>>> I'm currently trying to build test suites for openQA.
>>> One of the candidates is wget and, luckily, it already provides quite an
>>> extensive test suite.
>>> I have successfully built an RPM which has all that is needed for the tests.
>>> One test, Test-ftp-iri-fallback.px, fails on SLES12-SP2 and I can't see
>>> why.
>>
>> Look at tests/Test-ftp-iri-fallback.log, if you can't interpret the content
>> send it here.
>
> I cannot find any such file, no *.log" anywhere in the vicinity of the
> tests.
>
> Ah ... maybe I should have addede that I'm working on a slightly older
> version of wget: 1.14, which we ship with SLES12.
>
> NB I run the tests by calling
> run-px /var/opt/wget-tests
> The tests are installed in /var/opt/wget-tests/tests and the wget binary
> is in /var/opt/wget-tests/src (although I would have preferred to use
> the system's own wget, but that's a thing to be considered later).
>
> I want to run just the tests in an openQA environment to aid in
> integration testing. To that end, I am building an RPM with just enough
> to run the tests:
> tests/run-px
> tests/unit-tests
> tests/Test-*
> tests/*.pm
> tests/Makefile (not used)
> tests/WgetFeature.cfg
> tests/WgetTest.pm.in
> tests/certs/*
> src/wget
>
>>> Is there a list describing exactly what each test checks and what a
>>> failure means?
>>
>> Each test should self-contain a short description of it's purpose, sometimes
>> these are missing (accidentally).
>
> The accident must have happened here ;-)
> NB The only difference I find between the 1.14 and the 1.19 versions of
> this test is that the 1.19 version has the "name" hash tag in the
> "FTPTest->new()" call.
>
>> Test-ftp-iri-fallback tries to FTP-download a file containing non-ASCII
>> char(s).
>> The behavior of Wget (with IRI support) is to convert the file name to UTF-8
>> for using with a RETR command.
>> This should fail with a "550 file not found".
>
> It does.
>
>> Now Wget falls back to the unconverted file name and tries RETR again - this
>> should succeed (we told the FTP test server to know this file name).
>
> This indeed succeeds, but in the end:
>
> Test failed: file français.txt not downloaded
>
>
> As I cannot find any log file, here's the output the test produced
> (using cut-and-past from the ssh tty):
>
> Running Test-ftp-iri-fallback.px
>
> Running test Test-ftp-iri-fallback
> Calling ../src/wget --local-encoding=iso-8859-1 -S
> ftp://localhost:39938/français.txt
> --2017-06-09 16:42:53-- ftp://localhost:39938/fran%C3%A7ais.txt
>=> â<80><98>français.txtâ<80><99>
> Resolving localhost (localhost)... ::1, 127.0.0.1
> Connecting to localhost (localhost)|::1|:39938... failed: Connection
> refused.
> Connecting to localhost (localhost)|127.0.0.1|:39938... connected.
> Logging in as anonymous ...
> 220 GNU Wget Testing FTP Server ready.
> --> USER anonymous^M
>
> 230 Anonymous user access granted.
> --> SYST^M
>
> 215 UNIX Type: L8
> --> PWD^M
>
> 257 "/"
> --> TYPE I^M
>
> 200 TYPE changed to I.
> ==> CWD not needed.
> --> SIZE français.txt^M
>
> 550 File or directory not found.
> --> PASV^M
>
> 227 Entering Passive Mode (127,0,0,1,179,75)
> --> RETR français.txt^M
>
> Use of uninitialized value in string eq at
> /var/opt/wget-tests/tests/FTPServer.pm line 251, <$socket> chunk 7.
> 550 File not found.
>
> No such file â<80><98>français.txtâ<80><99>.
>
> --2017-06-09 16:42:53-- ftp://localhost:39938/fran%E7ais.txt
>=> â<80><98>français.txtâ<
Re: [Bug-wget] Timezones
On 07.03.2017 21:51, Mike Frysinger wrote: > On 07 Mar 2017 16:14, Josef Moellers wrote: >> When a colleague checked timestamps handling in vsftpd, he found that >> wget is expecting that timestamps received from ftp server have the same >> timezone as the client. >> >> in src/ftp-ls.c:ftp_parse_unix_ls() >> It reads time of modification from directory listing (.listing) which >> should be in UTC according to >> https://tools.ietf.org/html/rfc3659#section-3.1 and creates timestamp >> using mktime(timestruct) which honors local timezone configuration. This >> breaks timestamps. > > i don't see any of timezone (utc or otherwise) in that document. > iiuc, the time stamps are not standardized across OS's or server > implementations, so wget can't take a hard line either: it isn't > possible to always return the right answer. > -mike > Well ... The referenced section (which I should have read prior to forwarding this report) references a "time-val" and in section 2.3 of said document it explicitly states that "Time values are always represented in UTC (GMT), and in the Gregorian calendar regardless of what calendar may have been in use at the date and time indicated at the location of the server-PI." So: yes, you're right, the referenced section does not specify a time zone, but the RFC in general does. Josef
[Bug-wget] Timezones
When a colleague checked timestamps handling in vsftpd, he found that wget is expecting that timestamps received from ftp server have the same timezone as the client. in src/ftp-ls.c:ftp_parse_unix_ls() It reads time of modification from directory listing (.listing) which should be in UTC according to https://tools.ietf.org/html/rfc3659#section-3.1 and creates timestamp using mktime(timestruct) which honors local timezone configuration. This breaks timestamps. Josef
