Re: [Bug-wget] Unexpected wget -N behaviour for 1.17 onwards?
Hi Tim, Okay. Using the OpenSUSE-packaged wget (1.19.5) that comes with Leap 15.0: $ wget -r -N 192.168.2.100:8080 ... Reusing existing connection to 192.168.2.100:8080. HTTP request sent, awaiting response... 304 Not Modified File ‘192.168.2.100:8080/OaP6ysTyz6Y.mp 4’ not modified on server. Omitting download. This file is incomplete in my local copy. Trying again as you suggest, $ wget -r -N --no-if-modified-since 192.168.2.100:8080 ... --2019-02-10 08:35:14-- http://192.168.2.100:8080/OaP6ysTyz6Y.mp4 Reusing existing connection to 192.168.2.100:8080. HTTP request sent, awaiting response... 200 OK Length: 38044195 (36M) [application/octet-stream] The sizes do not match (local 8643456) -- retrieving. --2019-02-10 08:35:14-- http://192.168.2.100:8080/OaP6ysTyz6Y.mp4 Reusing existing connection to 192.168.2.100:8080. HTTP request sent, awaiting response... 200 OK Length: 38044195 (36M) [application/octet-stream] Saving to: ‘192.168.2.100:8080/OaP6ysTy z6Y.mp4 ... And it appears to work as expected. Won't this change to the behaviour of -N option subtly break a lot of scripts which rely on wget? Thanks so much, Tim. I do have an answer and a workaround though my concerns remain. Lawrence Wade Ottawa, Canada On Sun, Feb 10, 2019 at 2:11 AM Lawrence Wade wrote: > > Hi Everyone, > > This might be a corroboration of this > http://lists.gnu.org/archive/html/bug-wget/2018-10/msg00049.html > and this > https://bugs.launchpad.net/ubuntu/+source/wget/+bug/1715481 > > I use wget to backup my cellphone running Palapa Web Server, and it > has worked well for me for years. Since upgrading to OpenSUSE Leap 15, > I have been having corrupted files. > > My method is > $ wget -r -N 192.168.2.100:8080 > and if the connection is interrupted for any reason, the next time I > call wget it would complete any incomplete files. And since Leap 15, I > have been getting gradually corrupted backups. I was tearing my hair > out looking at wgetrc and other things. > > With one long file that I knew was incomplete, I got a Not Modified - > omitting download, even though I knew the file sizes were different > between the server and wget's copy - though the wget man page > explicitly states that if the file sizes do not match, -N will trigger > a download. > > I tried on OpenSUSE 42.3 (wget 1.14) and the incomplete file triggered > a download, even though wgetrc was identical. > > Again, on Leap 15, I compiled 1.20.1 (latest), 1.17.1, and then > finally with 1.16.3 the behaviour went back to what I expected (and I > got my corrupted phone backups fixed). > > Was a bug possibly introduced in 1.17 with the support for > --if-modified-since? > > Version shipping with OpenSUSE Leap 15: > GNU Wget 1.19.5 built on linux-gnu. > +cares +digest +gpgme +https +ipv6 +iri +large-file +metalink +nls > +ntlm +opie +psl +ssl/openssl > > Last version I tried where "wget -r -N" works as expected: > GNU Wget 1.16.3 built on linux-gnu. > +digest +https +ipv6 -iri +large-file +nls +ntlm +opie +psl +ssl/gnutls > > I'm open to the possibility that there may be something else causing > this bug, I have not found many mentions of it, but then again it is > subtle. You get pretty confident when you just let wget do its thing, > so there may be a lot of incomplete files out there... :) > > Thanks so much for your help. I can provide any other info that would > be helpful. > > Lawrence Wade > Ottawa, Canada
[Bug-wget] Unexpected wget -N behaviour for 1.17 onwards?
Hi Everyone, This might be a corroboration of this http://lists.gnu.org/archive/html/bug-wget/2018-10/msg00049.html and this https://bugs.launchpad.net/ubuntu/+source/wget/+bug/1715481 I use wget to backup my cellphone running Palapa Web Server, and it has worked well for me for years. Since upgrading to OpenSUSE Leap 15, I have been having corrupted files. My method is $ wget -r -N 192.168.2.100:8080 and if the connection is interrupted for any reason, the next time I call wget it would complete any incomplete files. And since Leap 15, I have been getting gradually corrupted backups. I was tearing my hair out looking at wgetrc and other things. With one long file that I knew was incomplete, I got a Not Modified - omitting download, even though I knew the file sizes were different between the server and wget's copy - though the wget man page explicitly states that if the file sizes do not match, -N will trigger a download. I tried on OpenSUSE 42.3 (wget 1.14) and the incomplete file triggered a download, even though wgetrc was identical. Again, on Leap 15, I compiled 1.20.1 (latest), 1.17.1, and then finally with 1.16.3 the behaviour went back to what I expected (and I got my corrupted phone backups fixed). Was a bug possibly introduced in 1.17 with the support for --if-modified-since? Version shipping with OpenSUSE Leap 15: GNU Wget 1.19.5 built on linux-gnu. +cares +digest +gpgme +https +ipv6 +iri +large-file +metalink +nls +ntlm +opie +psl +ssl/openssl Last version I tried where "wget -r -N" works as expected: GNU Wget 1.16.3 built on linux-gnu. +digest +https +ipv6 -iri +large-file +nls +ntlm +opie +psl +ssl/gnutls I'm open to the possibility that there may be something else causing this bug, I have not found many mentions of it, but then again it is subtle. You get pretty confident when you just let wget do its thing, so there may be a lot of incomplete files out there... :) Thanks so much for your help. I can provide any other info that would be helpful. Lawrence Wade Ottawa, Canada
Re: [Bug-wget] gnulib in wget needs an update
I've updated the gnulib version in the git repos. However, unless there is an actual problem with Wget crashing, the change will be available only with the next release. * LRN [190210 10:08]: > This[0] commit in gnulib fixed a critical bug that makes wget crash with > stackoverflow. The gnulib version in wget is slightly older than that. > > [0]: > https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=72e936e89c09bcf1a76479258881d91b0a27003f > -- Thanking You, Darshit Shah PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6 signature.asc Description: PGP signature
Re: [Bug-wget] Unexpected wget -N behaviour for 1.17 onwards?
Hi Lawrence, please try again with --no-if-modified-since. That should revert to the old behavior of using two requests (HEAD and then GET). If you still see issues, let us know. Some servers/proxies don't like the If-Modified-Since header. Regards, Tim On 10.02.19 08:11, Lawrence Wade wrote: > Hi Everyone, > > This might be a corroboration of this > http://lists.gnu.org/archive/html/bug-wget/2018-10/msg00049.html > and this > https://bugs.launchpad.net/ubuntu/+source/wget/+bug/1715481 > > I use wget to backup my cellphone running Palapa Web Server, and it > has worked well for me for years. Since upgrading to OpenSUSE Leap 15, > I have been having corrupted files. > > My method is > $ wget -r -N 192.168.2.100:8080 > and if the connection is interrupted for any reason, the next time I > call wget it would complete any incomplete files. And since Leap 15, I > have been getting gradually corrupted backups. I was tearing my hair > out looking at wgetrc and other things. > > With one long file that I knew was incomplete, I got a Not Modified - > omitting download, even though I knew the file sizes were different > between the server and wget's copy - though the wget man page > explicitly states that if the file sizes do not match, -N will trigger > a download. > > I tried on OpenSUSE 42.3 (wget 1.14) and the incomplete file triggered > a download, even though wgetrc was identical. > > Again, on Leap 15, I compiled 1.20.1 (latest), 1.17.1, and then > finally with 1.16.3 the behaviour went back to what I expected (and I > got my corrupted phone backups fixed). > > Was a bug possibly introduced in 1.17 with the support for > --if-modified-since? > > Version shipping with OpenSUSE Leap 15: > GNU Wget 1.19.5 built on linux-gnu. > +cares +digest +gpgme +https +ipv6 +iri +large-file +metalink +nls > +ntlm +opie +psl +ssl/openssl > > Last version I tried where "wget -r -N" works as expected: > GNU Wget 1.16.3 built on linux-gnu. > +digest +https +ipv6 -iri +large-file +nls +ntlm +opie +psl +ssl/gnutls > > I'm open to the possibility that there may be something else causing > this bug, I have not found many mentions of it, but then again it is > subtle. You get pretty confident when you just let wget do its thing, > so there may be a lot of incomplete files out there... :) > > Thanks so much for your help. I can provide any other info that would > be helpful. > > Lawrence Wade > Ottawa, Canada > signature.asc Description: OpenPGP digital signature
Re: [Bug-wget] gnulib in wget needs an update
On 10.02.2019 13:54, Darshit Shah wrote: > * LRN [190210 10:08]: >> This[0] commit in gnulib fixed a critical bug that makes wget crash with >> stackoverflow. The gnulib version in wget is slightly older than that. >> >> [0]: >> https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=72e936e89c09bcf1a76479258881d91b0a27003f >> > > I've updated the gnulib version in the git repos. However, unless there is an > actual problem with Wget crashing, the change will be available only with the > next release. > There is an actual problem with Wget crashing, when cross-compiled for a non-gnu, non-mingw host (in my case - Cygwin). The gl_cv_func_gettimeofday_clobber is guessed as "yes", which prompts NEED_LOCALTIME_BUFFER and REPLACE_LOCALTIME, and a wget compiled like that crashes on a simple http download. So this is somewhat rare, since few people cross-compile wget, but very real nevertheless. signature.asc Description: OpenPGP digital signature
