Re: [bug #64808] When I use wget to download some files from a web server, files with russian names do not get proper names
On 17 Nov 2023 at 20:34, grafgrim...@gmx.de wrote: Date sent: Fri, 17 Nov 2023 20:34:37 +0100 From: grafgrim...@gmx.de To: bug-wget@gnu.org Subject:Re: [bug #64808] When I use wget to download some files from a web server, files with russian names do not get proper names > I use Linux and so not exe files. I use Gentoo Linux. > > Command line example: > One line (wget and the url): > > wget > http://releases.mozilla.org/pub/firefox/releases/119.0.1/source/firefox-119.0.1.source.tar.xz > > result: a file with a wrong checksum. > > Greetings > Graf Grimm > What exact error are you seeing? Downloaded with wget and wget2 and worked fine? Downloaded wget http://releases.mozilla.org/pub/firefox/releases/119.0.1/source/firefox-119.0.1.source.tar.xz --2023-11-18 08:27:18-- http://releases.mozilla.org/pub/firefox/releases/119.0.1/source/firefox-119.0.1.source.tar.xz Resolving releases.mozilla.org (releases.mozilla.org)... 34.117.35.28 Connecting to releases.mozilla.org (releases.mozilla.org)|34.117.35.28|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 524717896 (500M) [application/x-tar] Saving to: ‘firefox-119.0.1.source.tar.xz’ firefox-119.0.1.source.tar.xz 100%[=>] 500.41M 5.89MB/s in 83s 2023-11-18 08:28:42 (6.00 MB/s) - ‘firefox-119.0.1.source.tar.xz’ saved [524717896/524717896] Moved file to subdirectory zz and then tar -xvf file no issues? Downloaded with wget2 [root@setzconote ~]# wget2 http://releases.mozilla.org/pub/firefox/releases/119.0.1/source/firefox-119.0.1.source.tar.xzfirefox-119.0.1.sour 100% [=>] 500.40M6.08MB/s [Files: 1 Bytes: 500.40M [5.94MB/s] Redirects: 0 Todo: 0 Errors: 0 ] [root@setzconote ~]# cmp firefox-119.0.1.source.tar.xz zz/firefox-119.0.1.source.tar.xz no difference in files. > On Fri, 17 Nov 2023 14:12:28 -0500 (EST) > invalid.nore...@gnu.org wrote: > > Follow-up Comment #4, bug #64808 (project wget): > > > > Windows character encodings may be special. > > Is this issue reproducible on e.g. GNU/Linux? > > I am willing to test it on GNU/Linux, but I need a full command line > > example from you. > > > > Out of curiosity, can you test wget2.exe from > > https://gitlab.com/gnuwget/wget2/-/releases (there is also a .sig > > file / PGP signature in case you want to verify the origin). > > > > > > > > ___ > > > > Reply to this item at: > > > > <https://savannah.gnu.org/bugs/?64808> > > > > _______________ > > Message sent via Savannah > > https://savannah.gnu.org/ > > > > > > > ++ Michael D. Setzer II - Computer Science Instructor (Retired) mailto:mi...@guam.net mailto:msetze...@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ ++
Re: fail to download big files correctly
On 16 Nov 2023 at 13:45, grafgrim...@gmx.de wrote: Date sent: Thu, 16 Nov 2023 13:45:42 +0100 From: grafgrim...@gmx.de To: bug-wget@gnu.org Subject:fail to download big files correctly > Dear wget developers, > > I use "GNU Wget 1.21.4" and I have problems to download big files with > wget. Unsure, what "big" exactly means. > > For example I download firefox-source, noto-font, linux-firmware and > get checksum failures when using wget. These files are 340 MB to 1 GB. > > size in bytes: > 356057052 linux-firmware-2023.tar.xz > 524717896 firefox-119.0.1.source.tar.xz > 1062488324 noto-20231031.tar.gz > > Download is okay when using a web browser or curl. > > So my weak guess is that wget can not handle big file downloads. > When I use wget several times to get the file, i always get another > checksum after download finished with wget. > > No problems when downloading small files. Unsure, what "small" exactly > means. Not sure on windows version, but tested downloading a 2099451904 Fedora-Workstation-Live-x86_64-38-1.6.iso using linux wget and wget2 programs. Both downloaded files on my local 1G network and both times files matched 100% with a binary compare. Downloaded Compared deleted fie and repeated. Got about 90MB download speed. Do know the wget2 had an issue with multi-threading with the 2.0 version, but the 2.1 version seems to handle up to 10 threads without issues. Only time I've used it with large files, and only locally with my local machines. Only have a 50Mb download and 3Mb upload, so not wanting to test. There is a windows build with the 2.1 version https://gitlab.com/gnuwget/wget2/-/releases Has 64 bit version for windows. > > Greetings > Graf Grimm > ++ Michael D. Setzer II - Computer Science Instructor (Retired) mailto:mi...@guam.net mailto:msetze...@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ ++
Building wget2 2.1.0 using docker.
/home/tim/s
rc/wget2/x86_64-w64-mingw32/lib/pkgconfig/libbrotlicommon.pc
./f298cc85f01aa7ee6b8b97809dd3296d412f2f9110d43d715a3f122ec14b9021/diff/home/tim/s
rc/wget2/x86_64-w64-mingw32/lib/libbrotlicommon.a
./f298cc85f01aa7ee6b8b97809dd3296d412f2f9110d43d715a3f122ec14b9021/diff/home/tim/s
rc/wget2/x86_64-w64-mingw32/lib/libbrotlienc.a
./f298cc85f01aa7ee6b8b97809dd3296d412f2f9110d43d715a3f122ec14b9021/diff/home/tim/s
rc/wget2/x86_64-w64-mingw32/lib/libbrotlidec.a
./f298cc85f01aa7ee6b8b97809dd3296d412f2f9110d43d715a3f122ec14b9021/diff/home/tim/s
rc/wget2/x86_64-w64-mingw32/bin
./f298cc85f01aa7ee6b8b97809dd3296d412f2f9110d43d715a3f122ec14b9021/diff/home/tim/s
rc/wget2/x86_64-w64-mingw32/bin/brotli
But had copied wget2.exe earlier to /root so have copy.
killall dockerd
since no longer needed?
Files in root created by process
24993339 Oct 27 04:44 wget2.exe
strip wget2.exe
# ls -l wget2.exe
4537344 Oct 27 04:58 wget2.exe
upx wget2.exe
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2023
UPX 4.1.0 Markus Oberhumer, Laszlo Molnar & John ReiserAug 8th 2023
File size Ratio Format Name
-- --- ---
4537344 -> 2386432 52.60%win64/pe wget2.exe
Packed 1 file.
# ls -l wget2.exe
2386432 Oct 27 04:58 wget2.exe
+------------+
Michael D. Setzer II - Computer Science Instructor (Retired)
mailto:mi...@guam.net
mailto:msetze...@gmail.com
Guam - Where America's Day Begins
G4L Disk Imaging Project maintainer
http://sourceforge.net/projects/g4l/
++
Re: How?
On 25 Oct 2023 at 13:44, ge...@mweb.co.za wrote: Date sent: Wed, 25 Oct 2023 13:44:27 +0200 (SAST) From: "ge...@mweb.co.za" To: Fernando Cassia Copies to: ENG WKJC , bug-wget Subject:Re: How? > Hi, > > thanks for that link again, Fernando (I had misplaced my note on it:-) > > And now I wonder if anyone has ported wget2 to Windows? > > Thanks, > > Gerd The wget2 site has the older 2.0.1 wget with a windows exe file, but I've been able to create the 2.1.0 version as well with info using docker? Seems to support up to 10 threads, which the 2.0.1 version doesn't seem to. https://gitlab.com/gnuwget/wget2/-/releases Built the wget2 using the docker, and it creates a wget2.exe file. 24993339 Oct 16 16:36 wget2.exe resulted file 7137339 Oct 16 16:37 wget2x.exe (compressed with upx). GNU Wget2 2.1.0 - multithreaded metalink/file/website downloader +digest +https +ssl/gnutls +ipv6 +iri +large-file -nls -ntlm -opie +psl -hsts -iconv +idn2 +zlib -lzma -brotlidec -zstd -bzip2 -lzip +http2 -gpgme Copyright (C) 2012-2015 Tim Ruehsen Copyright (C) 2015-2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://www.gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please send bug reports and questions to . Get same results as with the linux wget2 program. > > > > - Original Message - > From: "Fernando Cassia" > To: "ENG WKJC" > Cc: "bug-wget" > Sent: Wednesday, October 25, 2023 1:14:17 AM > Subject: Re: How? > > On Tue, 24 Oct 2023, 17:16 ENG WKJC, wrote: > > > Guys, > > > > Been looking for a solution that can run on Win7 or higher for HTTPS > > downloading. However, I don't understand the whole GNU thing, other than > > it's > > open source and generally free software. > > I've looked at the software links and don't know what to do here. > > > HI Marv > > You can find pre-built versions of wget for Windows on this Web site > > 32bit > https://eternallybored.org/misc/wget/1.21.4/32/wget.exe > > And 64 bit > https://eternallybored.org/misc/wget/1.21.4/64/wget.exe > > You just download the exe required for your system. > > How to tell if your Windows installation is 32bit or 64bit > https://support.microsoft.com/en-us/windows/32-bit-and-64-bit-windows-frequently-asked-questions-c6ca9541-8dce-4d48-0415-94a3faa2e13d > > ... then manually copy to a folder in your computer, and then place that > folder in your "path" (the path is a listing of folder locations that > Windows uses to look for exe files) > > (I personally use c:\utils and place all command line utilities there). > > --- > From the command prompt: > MD C:\Utils > Copy %userprofile%\downloads\wget.exe c:\utils > --- > > Here is a tutorial on how to the add that folder you ju St created to the > system path > > https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/ > > You can then run wget from the command prompt from any directory or folder > just by calling it with 'wget' > > Eg to get a listing of all options available > wget - -help > > Others will be able to guide you after that. > > If you want to see a video, sometimes a picture is better than a thousand > words > > https://youtu.be/cvvcG1a7dOM?si=Q5hO2GzlFFG0oRhC > > Best, > > FC > Buenos Aires, Argentina > ++ Michael D. Setzer II - Computer Science Instructor (Retired) mailto:mi...@guam.net mailto:msetze...@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ ++
Re: wget claims "Success" when it failed to write to local directory
On 21 Oct 2023 at 19:22, Tim Rühsen wrote: Date sent: Sat, 21 Oct 2023 19:22:25 +0200 Subject:Re: wget claims "Success" when it failed to write to local directory To: Christian Rosentreter , bug-wget@gnu.org From: Tim Rühsen Did a test with wget2 and get this result with error (2) if no write permission? wget2 https://www.christianrosentreter.com/ Failed to open 'index.html' (2) index.html 100% [== ===>]1.89K--.-KB/s [Files: 1 Bytes: 1.89K [2.23KB/s] Redirects: 0 Todo: 0 Errors: 0] With write permission it downloads index.html fine. wget2 https://www.christianrosentreter.com/ index.html 100% [== ==>]1.89K--.-KB/s [Files: 1 Bytes: 1.89K [2.27KB/s] Redirects: 0 Todo: 0 Errors: 0 ] > Hi, > > do you run the latest wget (1.21.4)? > > With that version, you get a > >Cannot write to 'index.html' (Permission denied). > > Regards, Tim > > On 10/21/23 17:16, Christian Rosentreter wrote: > > > > Hi there, > > > > There's a minor cosmetic bug in wget 1.x where it claims "Success" when it > > in fact entirely failed to write to the local > > disk, e.g. because of missing permissions/ write access to the current > > directory. The return code is "3" however, so it's > > basically only the message that it prints on screen that is funny in a > > suspicious way: > > > > > > ### Prepare situation… > > $ mkdir foobar > > $ chmod -w foobar # remove write access > > $ cd foobar > > > > > > ### Note: the "Permission denied" and "Cannot write to" messages, but we > > ### get a "(Success)" anyway: > > $ wget https://www.christianrosentreter.com/ > > --2023-10-21 17:05:35-- https://www.christianrosentreter.com/ > > Resolving www.christianrosentreter.com (www.christianrosentreter.com)... > > 85.13.142.16 > > Connecting to www.christianrosentreter.com > > (www.christianrosentreter.com)|85.13.142.16|:443... connected. > > HTTP request sent, awaiting response... 200 OK > > Length: unspecified [text/html] > > index.html: Permission denied > > > > Cannot write to 'index.html' (Success). > > > > > > ### The return code seems to be reasonable though: > > $ echo $? > > 3 > > > > > > ### Version tested: > > $ wget --version > > GNU Wget 1.21.4 built on darwin14.5.0. > > > > +cares +digest -gpgme +https +ipv6 +iri +large-file +metalink -nls > > +ntlm +opie +psl +ssl/openssl > > > > …cut… > > > > > > > > > > ++ Michael D. Setzer II - Computer Science Instructor (Retired) mailto:mi...@guam.net mailto:msetze...@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ ++
Re: wget claims "Success" when it failed to write to local directory
On 21 Oct 2023 at 17:16, Christian Rosentreter wrote: From: Christian Rosentreter Subject:wget claims "Success" when it failed to write to local directory Date sent: Sat, 21 Oct 2023 17:16:44 +0200 To: bug-wget@gnu.org > > Hi there, > > There's a minor cosmetic bug in wget 1.x where it claims "Success" when it in > fact entirely failed to write to the local > disk, e.g. because of missing permissions/ write access to the current > directory. The return code is "3" however, so it's > basically only the message that it prints on screen that is funny in a > suspicious way: > I'm a user of wget2 but in looking at man pages reports that error code 3 is a File I/O error? Exit Status Wget2 may return one of several error codes if it encounters problems. 0 No problems occurred. 1 Generic error code. 2 Parse error. For instance, when parsing command-line options, the .wget2rc or .netrc... 3 File I/O error. So, would agree the Success is probable an incorrect message, but since error code did report that there was a File I/O error the wget program worked successfully. > > ### Prepare situation… > $ mkdir foobar > $ chmod -w foobar # remove write access > $ cd foobar > > > ### Note: the "Permission denied" and "Cannot write to" messages, but we > ### get a "(Success)" anyway: > $ wget https://www.christianrosentreter.com/ > --2023-10-21 17:05:35-- https://www.christianrosentreter.com/ > Resolving www.christianrosentreter.com (www.christianrosentreter.com)... > 85.13.142.16 > Connecting to www.christianrosentreter.com > (www.christianrosentreter.com)|85.13.142.16|:443... connected. > HTTP request sent, awaiting response... 200 OK > Length: unspecified [text/html] > index.html: Permission denied > > Cannot write to 'index.html' (Success). > > > ### The return code seems to be reasonable though: > $ echo $? > 3 > > > ### Version tested: > $ wget --version > GNU Wget 1.21.4 built on darwin14.5.0. > > +cares +digest -gpgme +https +ipv6 +iri +large-file +metalink -nls > +ntlm +opie +psl +ssl/openssl > > …cut… > > > > > ++ Michael D. Setzer II - Computer Science Instructor (Retired) mailto:mi...@guam.net mailto:msetze...@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ ++
Issue with multi-tasking in wget2.exe
Sent a message to Tim Rühsen and got reply from Tim Rühsen (@rockdaboot) That said to sent email to this link. Have a program that uses wget2 under linux and it has no problems with using 13 threads to download 13 pages. Have found that in test with wine it has failures with threads set above 2, but they vary. Also, did some earlier test on windows 10 machine, and with native windows it wasn't getting errors till higher tests, but was doing whole process manually on windows.. Under Fedora 35 I ran this test script. for a in 1 2 3 4 5 6 7 8 9 10 11 12 13; do mkdir $a cd $a time wine wget2 --restrict-file-names=windows --max-threads=$a --secure-protocol=PFS -q --base="https://ssb-prod.ec.guamcc.edu/PROD/; -i ../testlist.sav cd .. done cd 1 for b in 2 3 4 5 6 7 8 9 10 11 12 13; do for a in *; do echo $a; cmp $a ../$b/$a ; done; done>>../output 2>../output2 Conents of testlist.sav bwpkedir.P_NameDirectory?dir_indx=_row=001_row=000=N= bwpkedir.P_NameDirectory?dir_indx=_row=026_row=025=N= bwpkedir.P_NameDirectory?dir_indx=_row=051_row=050=N= bwpkedir.P_NameDirectory?dir_indx=_row=076_row=075=N= bwpkedir.P_NameDirectory?dir_indx=_row=101_row=100=N= bwpkedir.P_NameDirectory?dir_indx=_row=126_row=125=N= bwpkedir.P_NameDirectory?dir_indx=_row=151_row=150=N= bwpkedir.P_NameDirectory?dir_indx=_row=176_row=175=N= bwpkedir.P_NameDirectory?dir_indx=_row=201_row=200=N= bwpkedir.P_NameDirectory?dir_indx=_row=226_row=225=N= bwpkedir.P_NameDirectory?dir_indx=_row=251_row=250=N= bwpkedir.P_NameDirectory?dir_indx=_row=276_row=275=N= bwpkedir.P_NameDirectory?dir_indx=_row=301_row=300=N= The times don't vary much with 1 task taking about 7 seconds, and 2 being about 6. Above 2 the times tend to take longer, except for the ones with failures where those pages are not downloded completely. The output2 shows these errors. 1, 2, 4, and 5 seemed to download all files the same, but others had differences. Only ran once, so not sure if results would always be same. Running under linux files are always identicle regardless of multi-task. cmp: EOF on ../3/bwpkedir.P_NameDirectory%3Fdir_indx=_row=026_row=025=N= after byte 24464, in line 544 cmp: EOF on ../6/bwpkedir.P_NameDirectory%3Fdir_indx=_row=001_row=000=N= after byte 15816, in line 322 cmp: EOF on ../7/bwpkedir.P_NameDirectory%3Fdir_indx=_row=026_row=025=N= after byte 15816, in line 319 cmp: EOF on ../8/bwpkedir.P_NameDirectory%3Fdir_indx=_row=001_row=000=N= after byte 8080, in line 125 cmp: EOF on ../8/bwpkedir.P_NameDirectory%3Fdir_indx=_row=126_row=125=N= after byte 8080, in line 121 cmp: EOF on ../9/bwpkedir.P_NameDirectory%3Fdir_indx=_row=201_row=200=N= after byte 32656, in line 756 cmp: EOF on ../10/bwpkedir.P_NameDirectory%3Fdir_indx=_row=076_row=075=N= after byte 8080, in line 125 cmp: EOF on ../10/bwpkedir.P_NameDirectory%3Fdir_indx=_row=176_row=175=N= after byte 8080, in line 125 cmp: EOF on ../11/bwpkedir.P_NameDirectory%3Fdir_indx=_row=126_row=125=N= after byte 8080, in line 121 cmp: ../11/bwpkedir.P_NameDirectory%3Fdir_indx=_row=176_row=175=N=: No such file or directory cmp: EOF on ../12/bwpkedir.P_NameDirectory%3Fdir_indx=_row=126_row=125=N= after byte 8080, in line 121 cmp: EOF on ../12/bwpkedir.P_NameDirectory%3Fdir_indx=_row=276_row=275=N= after byte 8080, in line 118 cmp: EOF on ../13/bwpkedir.P_NameDirectory%3Fdir_indx=_row=026_row=025=N= after byte 15816, in line 319 cmp: EOF on ../13/bwpkedir.P_NameDirectory%3Fdir_indx=_row=126_row=125=N= after byte 32656, in line 740 cmp: EOF on ../13/bwpkedir.P_NameDirectory%3Fdir_indx=_row=201_row=200=N= after byte 16211, in line 335 Windows 10 machine is current disconnected, so haven't done tests on a native windows. Last time, the errors didn't show till higher numbers. Once got up to 10 threads with no errors, but then ran again at 10, and had errors. With the windows native it was a manual process since scripting doesn't work as nice. +--------+ Michael D. Setzer II - Computer Science Instructor (Retired) mailto:mi...@guam.net mailto:msetze...@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ ++
