Re: fail to download big files correctly

[Jeffrey Walton]

On Fri, Nov 17, 2023 at 6:29 PM Tim Rühsen  wrote:
>
> On 11/17/23 20:34, grafgrim...@gmx.de wrote:
>  > I use Linux and so not exe files. I use Gentoo Linux.
>  >
>  > Command line example:
>  > One line (wget and the url):
>  >
>  > wget
>  >
> http://releases.mozilla.org/pub/firefox/releases/119.0.1/source/firefox-119.0.1.source.tar.xz
>  >
>  > result: a file with a wrong checksum.
>
> Just a guess:
>
> If you have a bad network and your connection drops, wget does retries
> by default.
> These retries may result in multiple incomplete files, so that the
> checksums are different. Can you do a 'ls -la' to see which size these
> files have?
>
> I currently can't simulate it - none of the "bad network" emulators for
> Linux do random connection drops.

This one has always made me laugh:
. If it is as bad as it sounds,
then you should be able to experience a dropped connection without
unplugging your ethernet cable.

(Comcast has a bad reputation in the US. I experienced it first hand
in the paqst).

Jeff

Re: fail to download big files correctly

[Tim Rühsen]

On 11/17/23 20:34, grafgrim...@gmx.de wrote:
> I use Linux and so not exe files. I use Gentoo Linux.
>
> Command line example:
> One line (wget and the url):
>
> wget
> 
http://releases.mozilla.org/pub/firefox/releases/119.0.1/source/firefox-119.0.1.source.tar.xz

>
> result: a file with a wrong checksum.

Just a guess:

If you have a bad network and your connection drops, wget does retries 
by default.
These retries may result in multiple incomplete files, so that the 
checksums are different. Can you do a 'ls -la' to see which size these 
files have?


I currently can't simulate it - none of the "bad network" emulators for 
Linux do random connection drops.


Regards, Tim


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: fail to download big files correctly

[Tim Rühsen]

On 11/16/23 13:45, grafgrim...@gmx.de wrote:

Dear wget developers,

I use "GNU Wget 1.21.4" and I have problems to download big files with
wget. Unsure, what "big" exactly means.

For example I download firefox-source, noto-font, linux-firmware and
get checksum failures when using wget. These files are 340 MB to 1 GB.

size in bytes:
356057052  linux-firmware-2023.tar.xz
524717896  firefox-119.0.1.source.tar.xz
1062488324 noto-20231031.tar.gz

Download is okay when using a web browser or curl.

So my weak guess is that wget can not handle big file downloads.
When I use wget several times to get the file, i always get another
checksum after download finished with wget.

No problems when downloading small files. Unsure, what "small" exactly
means.


I can't reproduce with the 1.21.4 version from Debian testing.

Can you send the output of 'wget --version'?
Where did you get the wget binary from?
Are you using a proxy? If yes, can you provide name and version?
Can you provide full links to the test files?
And anything else that let us reproduce your issue.



Greetings
Graf Grimm



Regards, Tim


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: fail to download big files correctly

[Michael D. Setzer II]

On 16 Nov 2023 at 13:45, grafgrim...@gmx.de wrote:

Date sent:  Thu, 16 Nov 2023 13:45:42 +0100
From:   grafgrim...@gmx.de
To: bug-wget@gnu.org
Subject:fail to download big files correctly

> Dear wget developers,
> 
> I use "GNU Wget 1.21.4" and I have problems to download big files with
> wget. Unsure, what "big" exactly means.
> 
> For example I download firefox-source, noto-font, linux-firmware and
> get checksum failures when using wget. These files are 340 MB to 1 GB.
> 
> size in bytes:
> 356057052  linux-firmware-2023.tar.xz
> 524717896  firefox-119.0.1.source.tar.xz
> 1062488324 noto-20231031.tar.gz
> 
> Download is okay when using a web browser or curl.
> 
> So my weak guess is that wget can not handle big file downloads.
> When I use wget several times to get the file, i always get another
> checksum after download finished with wget.
> 
> No problems when downloading small files. Unsure, what "small" exactly
> means.

Not sure on windows version, but tested downloading a 

2099451904 Fedora-Workstation-Live-x86_64-38-1.6.iso
using linux wget and wget2 programs.
Both downloaded files on my local 1G network and both times files 
matched 100% with a binary compare. Downloaded Compared
deleted fie and repeated. Got about 90MB download speed.

Do know the wget2 had an issue with multi-threading with the 2.0 
version, but the 2.1 version seems to handle up to 10 threads 
without issues. 

Only time I've used it with large files, and only locally with my 
local machines. Only have a 50Mb download and 3Mb upload, so 
not wanting to test. 

There is a windows build with the 2.1 version 
https://gitlab.com/gnuwget/wget2/-/releases
Has 64 bit version for windows.

> 
> Greetings
> Graf Grimm
> 


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

RE: fail to download big files correctly

[gerdd]

But then 1.21.4 should be the latest, at least on windows, according to 
eternallybored.GerdSent from my Galaxy
 Original message From: gerdd  Date: 
2023/11/16  15:35  (GMT+02:00) To: grafgrim...@gmx.de, bug-wget@gnu.org 
Subject: RE: fail to download big files correctly Not sure what hit you there. 
I once had the problem on windows with files greater than 4GB. Downloading the 
latest version of wget fixed that for me. Gerd.Sent from my Galaxy 
Original message From: grafgrim...@gmx.de Date: 2023/11/16  15:16  
(GMT+02:00) To: bug-wget@gnu.org Subject: fail to download big files correctly 
Dear wget developers,I use "GNU Wget 1.21.4" and I have problems to download 
big files withwget. Unsure, what "big" exactly means.For example I download 
firefox-source, noto-font, linux-firmware andget checksum failures when using 
wget. These files are 340 MB to 1 GB.size in bytes:356057052  
linux-firmware-2023.tar.xz524717896  
firefox-119.0.1.source.tar.xz1062488324 noto-20231031.tar.gzDownload is okay 
when using a web browser or curl.So my weak guess is that wget can not handle 
big file downloads.When I use wget several times to get the file, i always get 
anotherchecksum after download finished with wget.No problems when downloading 
small files. Unsure, what "small" exactlymeans.GreetingsGraf Grimm

RE: fail to download big files correctly

[gerdd]

Not sure what hit you there. I once had the problem on windows with files 
greater than 4GB. Downloading the latest version of wget fixed that for me. 
Gerd.Sent from my Galaxy
 Original message From: grafgrim...@gmx.de Date: 2023/11/16  
15:16  (GMT+02:00) To: bug-wget@gnu.org Subject: fail to download big files 
correctly Dear wget developers,I use "GNU Wget 1.21.4" and I have problems to 
download big files withwget. Unsure, what "big" exactly means.For example I 
download firefox-source, noto-font, linux-firmware andget checksum failures 
when using wget. These files are 340 MB to 1 GB.size in bytes:356057052  
linux-firmware-2023.tar.xz524717896  
firefox-119.0.1.source.tar.xz1062488324 noto-20231031.tar.gzDownload is okay 
when using a web browser or curl.So my weak guess is that wget can not handle 
big file downloads.When I use wget several times to get the file, i always get 
anotherchecksum after download finished with wget.No problems when downloading 
small files. Unsure, what "small" exactlymeans.GreetingsGraf Grimm