Re: Problems with progress on large files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Slava S. Kardakov wrote: Wget aborts on huge files: $ wget ftp://ftp.fsn.hu/testfiles/128T'http://bouncer.i-rs.ru/?product=OpenOffice.orglang=ruversion=2.3.1os=linuxinteltarball http://bouncer.i-rs.ru/?product=OpenOffice.orglang=ruversion=2.3.1os=linuxinteltarball' get: progress.c:972: create_image: Assertion `p - bp-buffer = bp-width' failed. Aborted But wget -q 'http://bouncer.i-rs.ru/?product=OpenOffice.orglang=ruversion=2.3.1os=linuxinteltarball http://bouncer.i-rs.ru/?product=OpenOffice.orglang=ruversion=2.3.1os=linuxinteltarball' works fine. I tried Wget 1.11 compiled with gcc 4.2.2 on 32-bit GNU/Linux system with glibc-2.7. Yeah; this was reported earlier today. Seems to be locale-dependent (if you set LC_ALL=C, it doesn't show up; at least not for the other reporter). - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHoD077M8hyUobTrERAn2RAJ0YQSk7BSRQf3iAPgJ5SRNLs9Q9egCggIyZ +W628VoPC4zmTv4CJbnph+U= =+GOo -END PGP SIGNATURE-
[wget] large memory allocation with -r and large files
Hi When downloading large files in web directory (directory listing enabled), with wget -r http://example.com/folder/ On Solaris with 512mb memory, wget fails with: wget: realloc: Failed to allocate 1073741824 bytes; memory exhausted. On Windows with 1gb memory, cygwin-wget uses too much memory (more than 2gb). Files are about 1gb each, and there're about 7-9 files. The same problem still persists if add other options are added. eg wget -r -c -np -nd -nH http://example.com/folder/ If wget was ran on just individual files, wget runs fine. eg. wget -O in http://example.com/folder/ wget -i in -F --base=http://example.com/folder/ http://example.com/folder/
FTP mirroring copy large files unnecessarily
Hi, First of all, I'm not on the list, so if you would cc me on any responses I would appreciate it. I'm using wget to mirror an ftp site. The problem I'm seeing is that very large files are copied each time, even though there are no changes. I couldn't find any known bugs related to this problem. I added some addition debugs to the source and it seems that when the size is larger than 7 digits, it losses track of the file size. After some poking around in the code, I came up with the following change that seems to solve the problem. 223,225c223,224 while (! ISDIGIT (*t) *t != 0) ++t; DEBUGP ((size: %s; , t)); --- while (t line ISDIGIT (*t)) --t; It seems from the comment that it's trying to work it's way back to the beginning of the size string, but t is already pointing to the beginning of the token. The problem that I think this was trying to address is the case where there is no space between the group name and the size. Instead, I changed the while look to traverse the string forward until it finds a digit. I'm not a developer, and I haven't written any serious c code in 10 years, so I may have this all wrong. The fact that I can't explain why the number if digits has any effect shows that I don't fully understand what's going on here. If anybody else can tell me another solution to mirror my ftp site without repeatedly copying all the largest files, I would appreciate it. -Tim Fletcher
Re: FTP mirroring copy large files unnecessarily
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tim Fletcher wrote: Hi, First of all, I'm not on the list, so if you would cc me on any responses I would appreciate it. I'm using wget to mirror an ftp site. The problem I'm seeing is that very large files are copied each time, even though there are no changes. I couldn't find any known bugs related to this problem. I added some addition debugs to the source and it seems that when the size is larger than 7 digits, it losses track of the file size. After some poking around in the code, I came up with the following change that seems to solve the problem. 223,225c223,224 while (! ISDIGIT (*t) *t != 0) ++t; DEBUGP ((size: %s; , t)); --- while (t line ISDIGIT (*t)) --t; Heh, you forgot to mention which file you were talking about! I've figured out that you mean ftp-ls.c Actually, someone else already reported this bug a couple months ago... Someone probably reported it before then, too, actually, since it was already fixed then as well: http://article.gmane.org/gmane.comp.web.wget.general/6997/match=ftp+ls+c The fix is already in place in Wget's development repository (http://hg.addictivecode.org/wget/1.11), and will be present in the next release of Wget, which is expected in a week or so (just waiting on some paperwork). - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHWbVg7M8hyUobTrERAi20AKCPPKHTCsgAAw2amxPgxNkIrrw13wCdEcJR CcFiygIrN0AX8w8II2J3rrQ= =M5i4 -END PGP SIGNATURE-
wget having trouble with large files
wget crashed when I tried to download a 5.9 GB file. I used http, but was redirected to ftp: $ wget 'http://www.archive.org/download/Bill_Gates_testimony_US_v_Microsoft_1998_video_p1/Bill_Gates_testimony_27081998.mpeg' --23:01:33-- http://www.archive.org/download/Bill_Gates_testimony_US_v_Microsoft_1998_video_p1/Bill_Gates_testimony_27081998.mpeg = `Bill_Gates_testimony_27081998.mpeg' Resolving www.archive.org... 207.241.233.58 Connecting to www.archive.org[207.241.233.58]:80... connected. HTTP request sent, awaiting response... 302 Found Location: ftp://ia310906.us.archive.org/1/items/Bill_Gates_testimony_US_v_Microsoft_1998_video_p1/Bill_Gates_testimony_27081998.mpeg [following] --23:01:34-- ftp://ia310906.us.archive.org/1/items/Bill_Gates_testimony_US_v_Microsoft_1998_video_p1/Bill_Gates_testimony_27081998.mpeg = `Bill_Gates_testimony_27081998.mpeg' Resolving ia310906.us.archive.org... 207.241.235.91 Connecting to ia310906.us.archive.org[207.241.235.91]:21... connected. Logging in as anonymous ... Logged in! == SYST ... done.== PWD ... done. == TYPE I ... done. == CWD /1/items/Bill_Gates_testimony_US_v_Microsoft_1998_video_p1 ... done. == PASV ... done.== RETR Bill_Gates_testimony_27081998.mpeg ... done. [ = ] -819,105,560 47.76K/s wget: retr.c:292: calc_rate: Assertion `bytes = 0' failed. Aborted (core dumped) $ ls -l Bill_Gates_testimony_27081998.mpeg -rw-rw-r-- 1 nisse staff 3475861736 Feb 16 23:01 Bill_Gates_testimony_27081998.mpeg Also note the wraparound, -819,105,560, in the progress display. I'm using wget-1.9.1, running on a (32-bit) x86, Debian GNU/Linux, sarge release. Regards, /Niels
Re: wget having trouble with large files
From: Niels Möller [...] I'm using wget-1.9.1, [...] You might try version 1.10.2, which offers large-file support. http://www.gnu.org/software/wget/wget.html Steven M. Schweda [EMAIL PROTECTED] 382 South Warwick Street(+1) 651-699-9818 Saint Paul MN 55105-2547
Re: Addition to problems with progress on large files
Thanks for the report. This patch fixes the problem:
2007-01-23 Hrvoje Niksic [EMAIL PROTECTED]
* progress.c (create_image): Check for ETA overflow.
(print_row_stats): Ditto.
Index: src/progress.c
===
--- src/progress.c (revision 2202)
+++ src/progress.c (working copy)
@@ -320,8 +320,10 @@
wgint bytes_remaining = dp-total_length - bytes_displayed;
/* The quantity downloaded in this download run. */
wgint bytes_sofar = bytes_displayed - dp-initial_length;
- int eta = (int) (dltime * bytes_remaining / bytes_sofar + 0.5);
- logprintf (LOG_VERBOSE, %s, eta_to_human_short (eta, true));
+ double eta = dltime * bytes_remaining / bytes_sofar;
+ if (eta INT_MAX - 1)
+ logprintf (LOG_VERBOSE, %s,
+ eta_to_human_short ((int) (eta + 0.5), true));
}
}
else
@@ -932,7 +934,10 @@
I found that doing that results in a very jerky and
ultimately unreliable ETA. */
wgint bytes_remaining = bp-total_length - size;
- eta = (int) (dl_total_time * bytes_remaining / bp-count + 0.5);
+ double eta_ = dl_total_time * bytes_remaining / bp-count;
+ if (eta_ = INT_MAX - 1)
+ goto skip_eta;
+ eta = (int) (eta_ + 0.5);
bp-last_eta_value = eta;
bp-last_eta_time = dl_total_time;
}
@@ -944,6 +949,7 @@
}
else if (bp-total_length 0)
{
+ skip_eta:
APPEND_LITERAL ( );
}
}
Problems with progress on large files
Wget aborts on huge files: $ wget ftp://ftp.fsn.hu/testfiles/128T get: progress.c:965: create_image: Assertion `p - bp-buffer = bp-width' failed. Aborted But wget -q ftp://ftp.fsn.hu/testfiles/128T works fine. I tried Wget 1.10+devel from svn trunk branch, Revision 2202. Compiled with gcc 3.4.6 on 32-bit GNU/Linux system with glibc-2.3.6. $ uname -a Linux slack 2.6.19.2 #1 Thu Jan 11 11:40:47 MSK 2007 i686 pentium4 i386 GNU/Linux -- With Best Regards, Stas. All You Need Is Love! Homo sum et nihil humani a me alienum puto.
Addition to problems with progress on large files
I've noticed problem is related with ETA calculating and on systems with high bandwidth it may not fail. It better to test something like $ wget --limit-rate=1024 ftp://ftp.fsn.hu/testfiles/128T -- With Best Regards, Stas. All You Need Is Love! Homo sum et nihil humani a me alienum puto.
Re: Downloading large files
[...] wget.exe version 1.8.2. You might try Wget 1.10.2, which has large-file support (where the underlying operating system and C run-time library do). http://www.gnu.org/software/wget/wget.html Steven M. Schweda (+1) 651-699-9818 382 South Warwick Street[EMAIL PROTECTED] Saint Paul MN 55105-2547
Re: Large files
Informatyk WFOŚiGW Rzeszów wrote: I cannot retrieve iso image from ftp server after broken connection on wget windows version 1.10.2. I receive 2.18GB but cannot no more. I execute wget with options: continue = on tries = 0 timeout = 180 passive_ftp = on -i -o could you please send us the error message printed by wget? -- Aequam memento rebus in arduis servare mentem... Mauro Tortonesi http://www.tortonesi.com University of Ferrara - Dept. of Eng.http://www.ing.unife.it GNU Wget - HTTP/FTP file retrieval tool http://www.gnu.org/software/wget Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net Ferrara Linux User Group http://www.ferrara.linux.it
problems with very large files
I tried wget ftp://sunsite.informatik.rwth-aachen.de/pub/comp/Linux/suse/i386/10.0/iso/SUSE-10.0-DVD-SRC-GM.iso which is a 4.7 GByte DVD image. wget ftp://sunsite.informatik.rwth-aachen.de/pub/comp/Linux/suse/i386/10.0/iso/SUSE-10.0-DVD-SRC-GM.iso --09:35:16-- ftp://sunsite.informatik.rwth-aachen.de/pub/comp/Linux/suse/i386/10.0/iso/SUSE-10.0-DVD-SRC-GM.iso = `SUSE-10.0-DVD-SRC-GM.iso' Resolving sunsite.informatik.rwth-aachen.de... 137.226.34.227 Connecting to sunsite.informatik.rwth-aachen.de[137.226.34.227]:21... connected. Logging in as anonymous ... Logged in! == SYST ... done.== PWD ... done. == TYPE I ... done. == CWD /pub/comp/Linux/suse/i386/10.0/iso ... done. == PASV ... done.== RETR SUSE-10.0-DVD-SRC-GM.iso ... done. Length: 398,651,392 (unauthoritative) 100%[] 581,101,544 908.91K/sETA 00:00 The size is incorrectly reported. Also the progress bar is incorrect after reaching the 400 MByte mark. The download seems to continue though. The version is GNU Wget 1.9.1. Cheers, Arne
Re: wget can't handle large files
Eberhard Wolff wrote: Hi, Apparently wget can't handle large file. hi eberhard, this is a known problem which was fixed in wget 1.10. please upgrade your wget installation. -- Aequam memento rebus in arduis servare mentem... Mauro Tortonesi http://www.tortonesi.com University of Ferrara - Dept. of Eng.http://www.ing.unife.it GNU Wget - HTTP/FTP file retrieval tool http://www.gnu.org/software/wget Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net Ferrara Linux User Group http://www.ferrara.linux.it
Re: wget can't handle large files
Am Dienstag, den 18.10.2005, 20:59 +0200 schrieb Eberhard Wolff: Apparently wget can't handle large file. There seems to be an overflow somewhere. See attached protocol: [G4:/Volumes/Large] wolff% wget --version GNU Wget 1.8.2 Large File support is available in wget 1.10 and later . You have to update your wget. -- Noèl Köthe noel debian.org Debian GNU/Linux, www.debian.org signature.asc Description: This is a digitally signed message part
RE: wget can't handle large files
Eberhard Wolff wrote: Apparently wget can't handle large file. [snip] wget --version GNU Wget 1.8.2 This bug was fixed in version 1.10 of wget. You should obtain a copy of the latest version, 1.10.2. Tony
Re: Bug when downloading large files (over 2 gigs) from proftpd server.
This problem has been fixed for the upcoming 1.10 release. If you want to try it, it's available at ftp://ftp.deepspace6.net/pub/ds6/sources/wget/wget-1.10-alpha2.tar.bz2
Bug when downloading large files (over 2 gigs) from proftpd server.
Hi,
When using wget (version 1.9.1 running on Debian Sarge) to download
files over 2 gigs from an ftp server (proftpd), wget reports a negative
length and keeps downloading, but once the file is successfully
downloaded it crashes (and therefore doesn't download the rest of the
files). Here is the output from wget:
server:/home# wget ftp://marjan/{homayoun,horace,marjan,msoleyma}.tar
--18:58:25-- ftp://marjan/homayoun.tar
= `homayoun.tar'
Resolving marjan... 192.168.0.1
Connecting to marjan[192.168.0.1]:21... connected.
Logging in as anonymous ... Logged in!
== SYST ... done.== PWD ... done.
== TYPE I ... done. == CWD not needed.
== PASV ... done.== RETR homayoun.tar ... done.
Length: -1,540,120,576 (unauthoritative)
[ =] -1,540,120,576 777.41K/s
wget: retr.c:292: calc_rate: Assertion `bytes = 0' failed.
Hope this helps,
Bijan Soleymani
Re: O_EXCL and large files
Maciej W. Rozycki [EMAIL PROTECTED] writes: Is it possible to portably use open() and retain large file support? Try the AC_SYS_LARGEFILE autoconf macro. That's what I thought I was using. I was just afraid that open() wasn't correctly encompassed by the large file API's, a fear that proved unfounded. You may also want AC_FUNC_FSEEKO for fseeko(). I wonder what is the difference between AC_FUNC_FSEEKO and AC_CHECK_FUNCS(seeko). The manual doesn't seem to explain. I was already a bit surprised having failed to see AC_SYS_LARGEFILE being called from your patch, Are you sure you looked at my patch? All three versions of the patch pretty much begin with the +AC_SYS_LARGEFILE change to configure.in.
Re: O_EXCL and large files
On Wed, 23 Feb 2005, Hrvoje Niksic wrote: You may also want AC_FUNC_FSEEKO for fseeko(). I wonder what is the difference between AC_FUNC_FSEEKO and AC_CHECK_FUNCS(seeko). The manual doesn't seem to explain. Well, that's what I have on my local system: - Macro: AC_FUNC_FSEEKO If the `fseeko' function is available, define `HAVE_FSEEKO'. Define `_LARGEFILE_SOURCE' if necessary to make the prototype visible on some systems (e.g. glibc 2.2). Otherwise linkage problems may occur when compiling with `AC_SYS_LARGEFILE' on largefile-sensitive systems where `off_t' does not default to a 64bit entity. The _LARGEFILE_SOURCE bit is the important one -- I have a vague memory of some program having a problem once in this area. I was already a bit surprised having failed to see AC_SYS_LARGEFILE being called from your patch, Are you sure you looked at my patch? All three versions of the patch pretty much begin with the +AC_SYS_LARGEFILE change to configure.in. Well, I must have been blind. Or I'm simply too busy these days. Anyway, sorry about the confusion and I'm glad you've got this right despite the issues with LFS being too messy on many systems with no good reason. Maciej
Re: O_EXCL and large files
Maciej W. Rozycki [EMAIL PROTECTED] writes: I wonder what is the difference between AC_FUNC_FSEEKO and AC_CHECK_FUNCS(seeko). The manual doesn't seem to explain. Well, that's what I have on my local system: - Macro: AC_FUNC_FSEEKO If the `fseeko' function is available, define `HAVE_FSEEKO'. Define `_LARGEFILE_SOURCE' if necessary to make the prototype visible on some systems (e.g. glibc 2.2). Otherwise linkage problems may occur when compiling with `AC_SYS_LARGEFILE' on largefile-sensitive systems where `off_t' does not default to a 64bit entity. The _LARGEFILE_SOURCE bit is the important one -- I have a vague memory of some program having a problem once in this area. I see. Now I know why I didn't use AC_FUNC_FSEEKO -- Wget doesn't use fseeko anywhere. The only usage we have is: fseek (fp, 0, SEEK_END); size = ftell (fp); /* replaced with ftello where available */ Is there need to use fseeko here?
Re: O_EXCL and large files
On Wed, 23 Feb 2005, Hrvoje Niksic wrote: I see. Now I know why I didn't use AC_FUNC_FSEEKO -- Wget doesn't use fseeko anywhere. The only usage we have is: fseek (fp, 0, SEEK_END); size = ftell (fp); /* replaced with ftello where available */ Is there need to use fseeko here? Probably -- the Single Unix Specification, Version 2, which introduced the o functions, quotes this error code: [EOVERFLOW] For fseek(), the resulting file offset would be a value which cannot be represented correctly in an object of type long. I don't know if it happens in practice (GNU libc seems to be forgiving, as usual). But we do want to use ftello(), so we need AC_FUNC_FSEEKO anyway as the _LARGEFILE_SOURCE restriction applies to both. This is what autoconf manual says (yes, a reference to ftello() should really be included in the AC_FUNC_FSEEKO description): The LFS introduced the `fseeko' and `ftello' functions to replace their C counterparts `fseek' and `ftell' that do not use `off_t'. Take care to use `AC_FUNC_FSEEKO' to make their prototypes available when using them and large-file support is enabled. And if using ftello(), then we should probably use fseeko() as well, for consistency if nothing else. Maciej
O_EXCL and large files
When opening files, Wget takes care (by default) to not overwrite an
existing file, and to tell the user where the file is to be saved.
However, the defense against overwriting may fail because Wget
determines the file name before attempting the download, but only
opens the file when the data starts to arrive. (This is intentional,
so that it doesn't leave behind empty files if the user interrupts the
program before data arrives.)
This can be fixed by using O_EXCL when opening the file. If the
exclusive open fails with EEXIST, we retry the download -- because we
can't download to the file name we promised we'd use. The code
could look like this:
FILE *
fopen_excl (const char *file)
{
#ifdef O_EXCL
int fd = open (file, O_CREAT | O_EXCL | O_WRONLY);
if (fd 0)
return NULL;
return fdopen (file, wb);
#else
return fopen (file, wb);
#endif
}
However, I suspect that this will not work correctly with large files.
strace shows that, when large files are used, Linux fopen includes
O_LARGEFILE among the flags. Solaris, on the other hand, seems to use
the open64 function. (But will open be automatically mapped to open64
when _FILE_OFFSET_BITS is 64?) For all I know, other systems may
require something different.
Is it possible to portably use open() and retain large file support?
Re: O_EXCL and large files
From: Hrvoje Niksic: [...] Solaris, on the other hand, seems to use the open64 function. (But will open be automatically mapped to open64 when _FILE_OFFSET_BITS is 64?) For all I know, other systems may require something different. SunOS 5.9 /usr/include/fcntl.h: [...] /* large file compilation environment setup */ #if !defined(_LP64) _FILE_OFFSET_BITS == 64 #ifdef __PRAGMA_REDEFINE_EXTNAME #pragma redefine_extname openopen64 #pragma redefine_extname creat creat64 [...] The idea is to continue to use open(), but to define the right macros to get the right open(). On VMS (with its RMS I/O layer), open() is not so fundamental, and only functions which explicitly use off_t seem to be affected. And yes, VMS does require something different. The macro there is _LARGEFILE. (But that's a problem for config.h and/or the builders.) A quick look at Tru64 UNIX suggests that large-file is all there is. I'd say that if it fails on Linux, then Linux has a problem. (But I'd expect it to be fine if you do it correctly.) Steven M. Schweda (+1) 651-699-9818 382 South Warwick Street[EMAIL PROTECTED] Saint Paul MN 55105-2547
Re: O_EXCL and large files
[EMAIL PROTECTED] (Steven M. Schweda) writes: SunOS 5.9 /usr/include/fcntl.h: [...] /* large file compilation environment setup */ #if !defined(_LP64) _FILE_OFFSET_BITS == 64 #ifdef __PRAGMA_REDEFINE_EXTNAME #pragma redefine_extnameopenopen64 #pragma redefine_extnamecreat creat64 [...] The idea is to continue to use open(), but to define the right macros to get the right open(). Good to know. Regarding Linux, it turns out I spoke too soon. Apparently it also has an open64, except it's not visible by trace because the wrapper is in libc. When I actually tried to run the above code snippet, the O_LARGEFILE flag was there all right.
GNU Wget 1.9.1 Issues with large files ( 2 GB)
Dear GNU, I was trying to download SuSE version 9.2 from the local mirror site thinking that I could get the entire package as a single DVD image ( 2 GB). So I did the wget command with the appropriate FTP arguments, and run it in the background. The first clue that this was going to have some problems was when I inspected the wget-log file and found this: == RETR SUSE-Linux-9.2-FTP-DVD.iso ... done. Length: -931,424,256 [-1,275,058,088 to go] [ skipping 335550K ] 335550K ,, ,, ,. .. ..-36% 25.24 KB/s The length and to go numbers as well as the -36% was interesting to me, but I figured that it was just a print format problem. But what I found was that when the file reached approximately 2 GB in size, wget just quit. No error message in the log, just quit. Trying to resume wget on that file appears to try and skip forward the amount downloaded, but then also just quit. Perhaps there is an issue with the ftp server and large files, or, more likely, that wget can't handle large files. Definitely worth some investigation I think. Let me know if I need to provide more information or testing for you. Thanks, Erik.
GNU Wget 1.9.1 Issues with large files ( 2 GB)
Dear GNU, I was trying to download SuSE version 9.2 from the local mirror site thinking that I could get the entire package as a single DVD image ( 2 = GB). So I did the wget command with the appropriate FTP arguments, and run it = in the background. The first clue that this was going to have some problems was when I inspected the wget-log file and found this: =3D=3D RETR SUSE-Linux-9.2-FTP-DVD.iso ... done. Length: -931,424,256 [-1,275,058,088 to go] [ skipping 335550K ] 335550K ,, ,, ,. .. ..-36% = 25.24 KB/s The length and to go numbers as well as the -36% was interesting to = me, but I figured that it was just a print format problem. But what I found = was that when the file reached approximately 2 GB in size, wget just quit. = No error message in the log, just quit. Trying to resume wget on that file appears to try and skip forward the amount downloaded, but then also = just quit. Perhaps there is an issue with the ftp server and large files, or, = more likely, that wget can't handle large files. Definitely worth some investigation I think. Let me know if I need to provide more information or testing for = you. Thanks, Erik.
Re: wget support for large files
On Feb 18, 2005, at 9:16 PM, Mauro Tortonesi wrote: On Saturday 12 February 2005 10:29 am, Chris Ross wrote: The wget web page at www.gnu.org has a link for the mailing list that doesn't work. So I'm emailing here. which link? could you please tell me so that i can fix it? Under Request an Enhancement there, is a link to: http://www.gnu.org/software/wget/mailinglists Which doesn't exist. Thanks... - Chris
Re: GNU Wget 1.9.1 Issues with large files ( 2 GB)
Erik Ohrnberger [EMAIL PROTECTED] writes: I was trying to download SuSE version 9.2 from the local mirror site thinking that I could get the entire package as a single DVD image ( 2 GB). So I did the wget command with the appropriate FTP arguments, and run it in the background. Unfortunately, Wget does not yet support downloading files larger than 2G. It is being worked on, though, and will most likely appear in the next version. Thanks for the report.
Re: wget support for large files
On Saturday 12 February 2005 10:29 am, Chris Ross wrote: The wget web page at www.gnu.org has a link for the mailing list that doesn't work. So I'm emailing here. which link? could you please tell me so that i can fix it? -- Aequam memento rebus in arduis servare mentem... Mauro Tortonesi University of Ferrara - Dept. of Eng.http://www.ing.unife.it Institute of Human Machine Cognition http://www.ihmc.us Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net Ferrara Linux User Group http://www.ferrara.linux.it
Bug: really large files cause problems with status text
When downloading a 4.2 gig file (such as from ftp://movies06.archive.org/2/movies/abe_lincoln_of_the_4th_ave/abe_lincoln_o f_the_4th_ave.mpeg ) cause the status text (i.e. 100%[+===] 38,641,328 213.92K/sETA 00:00) to print invalid things (in this case, that 100% of the file has been downloaded, even though only 40MB really has.
Re: Bug: really large files cause problems with status text
Quoting Alan Robinson [EMAIL PROTECTED]: When downloading a 4.2 gig file (such as from ftp://movies06.archive.org/2/movies/abe_lincoln_of_the_4th_ave/abe_lincoln_o f_the_4th_ave.mpeg ) cause the status text (i.e. 100%[+===] 38,641,328 213.92K/sETA 00:00) to print invalid things (in this case, that 100% of the file has been downloaded, even though only 40MB really has. It is a Frequently Asked Question, with the answer that people are working on it. // Ulf
wget has issues with large files (over 2GB)
First of all, many thanks for a rock-solid utility! OS: SuSE Linux 9.1: rpm -q wget wget-1.9.1-45 I am resuming the download of a DVD image but the sizes seem to overflow 32-bit signed integer. wget -c ftp://mirror.mcs.anl.gov/pub/suse/i386/9.2/iso/SUSE-Linux-9.2-FTP-DVD.iso == RETR SUSE-Linux-9.2-FTP-DVD.iso ... done. Length: -931,424,256 [-1,990,048,240 to go] (unauthoritative)
Re: wget has issues with large files (over 2GB)
On Tue, Jan 25, 2005 at 02:44:16PM +, Andrew Robb wrote: First of all, many thanks for a rock-solid utility! OS: SuSE Linux 9.1: rpm -q wget wget-1.9.1-45 I am resuming the download of a DVD image but the sizes seem to overflow 32-bit signed integer. wget -c ftp://mirror.mcs.anl.gov/pub/suse/i386/9.2/iso/SUSE-Linux-9.2-FTP-DVD.iso == RETR SUSE-Linux-9.2-FTP-DVD.iso ... done. Length: -931,424,256 [-1,990,048,240 to go] (unauthoritative) This could be a problem with the FTP server. What is the FTP server software? -- Ryan Underwood, [EMAIL PROTECTED]
large files
Hi I suggest: That you apply and commit the currently best patch for large files. That is probably the one Leonid works/worked on. Isn't that the one Redhat has applied? I suggest that off_t isn't used to store/keep large file sizes, but instead a privately typedefed type instead. Named wget_off_t for example. On linux and most unix-like systems, we can 'typedef off_t wget_off_t' and most things will be fine. A strtoll() replacement is needed for some platforms. I believe this works fine for VMS too. I could check later. On systems like Windows, large file support takes some more work, and then the wget_off_t is probably best made 'typedef signed __int64 wget_off_t' with MSVC (but all libc-calls using wget_off_t will need attention/replacements since the windows libc APIs aren't supporting large files). On Windows with mingw or watcom, the typedef is better made 'typedef long long wget_off_t'. On Windows CE, I believe it works best as 'typedef long wget_off_t'. On systems without large file support, off_t is most often still present and simply just a 32 bit type. Thus, a file-size-as-string-to-number function (basicly strtol() or strtoll() depending on the platform) is suitable, to convert a string to wget_off_t. Now, this is only a suggestion meant to kick-start some discussions and possibly implementations. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
wget bug with large files
I got a crash in wget downloading a large iso file (2,4 GB) newdeal:/pub/isos# wget -c ftp://ftp.belnet.be/linux/fedora/linux/core/3/i386/iso/FC3-i386-DVD.iso --09:22:17-- ftp://ftp.belnet.be/linux/fedora/linux/core/3/i386/iso/FC3-i386-DVD.iso = `FC3-i386-DVD.iso' Resolving ftp.belnet.be... 193.190.198.20 Connecting to ftp.belnet.be[193.190.198.20]:21... connected. Accesso come utente anonymous ... Login eseguito! == SYST ... fatto. == PWD ... fatto. == TYPE I ... fatto. == CWD /linux/fedora/linux/core/3/i386/iso ... fatto. == SIZE FC3-i386-DVD.iso ... fatto. == PASV ... fatto. == REST 2079173504 ... fatto. == RETR FC3-i386-DVD.iso ... fatto. 100%[+=] 2,147,470,560 60.39K/s ETA 00:00wget: progress.c:704: create_image: Assertion `insz = dlsz' failed. Aborted then I tried to resume the download .. newdeal:/pub/isos# wget -c ftp://ftp.belnet.be/linux/fedora/linux/core/3/i386/iso/FC3-i386-DVD.iso --09:41:40-- ftp://ftp.belnet.be/linux/fedora/linux/core/3/i386/iso/FC3-i386-DVD.iso = `FC3-i386-DVD.iso' Resolving ftp.belnet.be... 193.190.198.20 Connecting to ftp.belnet.be[193.190.198.20]:21... connected. Accesso come utente anonymous ... Login eseguito! == SYST ... fatto. == PWD ... fatto. == TYPE I ... fatto. == CWD /linux/fedora/linux/core/3/i386/iso ... fatto. == SIZE FC3-i386-DVD.iso ... fatto. == PASV ... fatto. == REST -2147476576 ... REST fallito, ricomincio dall'inizio. (restarting from beginning) == RETR FC3-i386-DVD.iso ... fatto. [ =] 551,648 63.87K/s Here it deleted the old iso image (2,1GB downloaded) and started from the beginning .. shouldn't it save the new file with a .1 suffix ? Let me know if I can help you tracking this bug Thanks, -- Roberto Sebastiano [EMAIL PROTECTED]
Wget and large files: Solution available
Hello everybody, I am using wget for automated database updates, but my files are now by far exceeding 2GB of size. As far as I could find out, wget has still problems with this file size limitation, right? At least the latest version I downloaded still does... Is there a solution foreseen in the near future? Are there any alternative programs to wget that I could use that do not suffer from this limitation??? Please include me personally in all replys, since I am not a subscriber of this list! Thanks a lot for any comments Maik
Re: Wget and large files: Solution available
Maik, As you could see from http://www.mail-archive.com/wget%40sunsite.dk/msg06652.html and other postings, this problem will be fixed in the next release of the official version. If you need to download files larger than 2Gb now, get patched, unofficial version of wget from http://software.lpetrov.net/wget-LFS/ Leonid
Re: Large Files Support for Wget
On Tue, 11 May 2004, Hrvoje Niksic wrote: That in itself is not a problem because, under Windows, off_t will be typedeffed to a 64-bit type if LFS is available, and to `int' otherwise. Sorry for my confusion, but when I think about it I believe Windows _do_ have an off_t, but that is merely 32bit. The plain libc in Windows seem to be limited to 32 bits, even though the actual underlying code support 64bit sizes. In curl we work around this by providing our own curl_off_t variable. The point of my question was: should low-level code even care whether LFS is in use (by using off_t for various variables), or should it use intmax_t to get the largest representation available on the system? The latter is in principle sort of like using `long', except you're not tied to the actual size of `long'. In curl we went with curl_off_t for various variables and that is then 32bit or 64bit depending on platform. I can't see many benefits in using 64bit variables on systems that don't deal with 64bit filesizes. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Large Files Support for Wget
Daniel Stenberg [EMAIL PROTECTED] writes: On Mon, 10 May 2004, [iso-8859-2] Dra?en Ka?ar wrote: * Change most (all?) occurrences of `long' in the code to `off_t'. Or should we go the next logical step and just use uintmax_t right away? Just use off_t. ... but Windows has no off_t... ;-) That in itself is not a problem because, under Windows, off_t will be typedeffed to a 64-bit type if LFS is available, and to `int' otherwise. The point of my question was: should low-level code even care whether LFS is in use (by using off_t for various variables), or should it use intmax_t to get the largest representation available on the system? The latter is in principle sort of like using `long', except you're not tied to the actual size of `long'.
RE: Large Files Support for Wget
-Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] * Profit! I think you'd really deserve some. Heiko -- -- PREVINET S.p.A. www.previnet.it -- Heiko Herold [EMAIL PROTECTED] [EMAIL PROTECTED] -- +39-041-5907073 ph -- +39-041-5907472 fax
Re: Large Files Support for Wget
Hrvoje Niksic wrote: David Fritz [EMAIL PROTECTED] writes: IIUC, GNU coreutils uses uintmax_t to store large numbers relating to the file system and prints them with something like this: char buf[INT_BUFSIZE_BOUND (uintmax_t)]; printf (_(The file is %s octets long.\n), umaxtostr (size, buf)); That's probably the most portable way to do it. For the time being. However, in C99 %ju is the correct format for printing uintmax_t. There are systems which have uintmax_t, but don't have the j modifier, so the whole thing is a problem if you want to write failsafe configure check. And there might be run-time problems, as well. * Change most (all?) occurrences of `long' in the code to `off_t'. Or should we go the next logical step and just use uintmax_t right away? Just use off_t. -- .-. .-.Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | |[EMAIL PROTECTED]
Re: Large Files Support for Wget
On Mon, 10 May 2004, [iso-8859-2] Dra?en Ka?ar wrote: * Change most (all?) occurrences of `long' in the code to `off_t'. Or should we go the next logical step and just use uintmax_t right away? Just use off_t. ... but Windows has no off_t... ;-) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Large Files Support for Wget
IIUC, GNU coreutils uses uintmax_t to store large numbers relating to the file system and prints them with something like this: char buf[INT_BUFSIZE_BOUND (uintmax_t)]; printf (_(The file is %s octets long.\n), umaxtostr (size, buf)); where umaxtostr() has the following prototype: char *umaxtostr (uintmax_t, char *); and it returns its second argument (the address of the buffer provided by the caller) so it can be used easily as an argument in printf calls.
Re: Large Files Support for Wget
David Fritz [EMAIL PROTECTED] writes: IIUC, GNU coreutils uses uintmax_t to store large numbers relating to the file system and prints them with something like this: char buf[INT_BUFSIZE_BOUND (uintmax_t)]; printf (_(The file is %s octets long.\n), umaxtostr (size, buf)); That's probably the most portable way to do it. I guess that solves the remaining technical difficulty. The things that need to be done for large file support then are: * Use AC_SYS_LARGEFILE in configure.in. Make sure the compilation flags contain the proper large file support incantations. * Change most (all?) occurrences of `long' in the code to `off_t'. Or should we go the next logical step and just use uintmax_t right away? * Profit! Some of this has already been done in the submitted patch.
Re: Large Files Support for Wget
[ Moving discussion to [EMAIL PROTECTED] ] Gisle Vanem [EMAIL PROTECTED] writes: Hrvoje Niksic [EMAIL PROTECTED] said: Supporting large files in a really portable manner is unfortunately not trivial. It's not that hard. Look at how we did this in libcurl; basically some define trickery; #define FILE_OFF_T 32/64 bit unsigned Fair enough. But isn't off_t signed? #define FILE_OFF_FMT %llu or %Lu How does gettext cope with that? For example, this string is what worries me: printf (_(The file is FILE_OFF_FMT octets long.\n), size);
Re: Large Files Support for Wget
Hrvoje Niksic [EMAIL PROTECTED] said: #define FILE_OFF_T 32/64 bit unsigned Fair enough. But isn't off_t signed? Obs, yes. A 'long' on Win32. #define FILE_OFF_FMT %llu or %Lu How does gettext cope with that? For example, this string is what worries me: printf (_(The file is FILE_OFF_FMT octets long.\n), size); I assume Wget needs a msg-entry for each string. The file is %Ld octets long.\n The file is %lld octets long.\n unless messages is built on same platform as Wget is. Not sure this is possible with the msg* tools. --gv
Re: Large Files Support for Wget
Gisle Vanem [EMAIL PROTECTED] writes: printf (_(The file is FILE_OFF_FMT octets long.\n), size); I assume Wget needs a msg-entry for each string. The file is %Ld octets long.\n The file is %lld octets long.\n The problem is, which of those message entries should end up in `wget.pot' and in translations? unless messages is built on same platform as Wget is. Even if messages are, translations are probably not.
wget bug in retrieving large files 2 gig
Hi, While downloading a file of about 3,234,550,172 bytes with wget http://foo/foo.mpg; I get an error: HTTP request sent, awaiting response... 200 OK Length: unspecified [video/mpeg] [ = ] -1,060,417,124 13.10M/s wget: retr.c:292: calc_rate: Assertion `bytes = 0' failed. Aborted The md5sum of downloaded and origanal file is de same! So there should not be an error. The amound of 'bytes downloaded' during is not correct also: It become negative over 2 gig. greetings from the Netherlands, Eduard
Re: wget crashes when working on large files
This is a bug -- regardless of whether it supports large files or not, Wget shouldn't crash when dealing with them. I plan to look into large file support for the next version of Wget.
wget crashes when working on large files
Hi!: I use wget almost everyday, is such a great tool! I found that present version (wget --version : GNU Wget 1.9 and previous one 1.8) crashes with large files: For example, trying to get a DVD ISO from the Gutenberg Project: ** nohup wget -c ftp://ftp.ibiblio.org/pub/docs/books/gutenberg/cdimages/pgdvd.iso A second bug (a little one), is that in the percentages for large files it gives negative values. i.e.: *** 1300K .. .. .. .. .. 0% 26.42 KB/s 1350K .. .. .. .. .. 0% 57.44 KB/s 1400K .. .. .. .. .. 0% 30.85 KB/s 1450K .. .. .. .. .. 0% 31.30 KB/s 1500K .. .. .. .. .. -1% 48.76 KB/s 1550K .. .. .. .. .. -1% 53.21 KB/s 1600K .. .. .. .. .. -1% 59.09 KB/s 1650K .. .. .. .. .. -1% 48.31 KB/s 1700K .. .. .. .. .. -1% 29.26 KB/s *** I guess this problem is related with the previous one. The length value is negative too, ie: *** --14:24:11-- ftp://ftp.ibiblio.org/pub/docs/books/gutenberg/cdimages/pgdvd.iso = `pgdvd.iso' Resolving ftp.ibiblio.org... 152.2.210.81 Connecting to ftp.ibiblio.org[152.2.210.81]:21... connected. Logging in as anonymous ... Logged in! == SYST ... done.== PWD ... done. == TYPE I ... done. == CWD /pub/docs/books/gutenberg/cdimages ... done. == PORT ... done.== RETR pgdvd.iso ... done. Length: -155,320,320 (unauthoritative) * And finally, when trying to get the rest of the file it crashes, ie: *** dvd]$ wget -c ftp://ftp.ibiblio.org/pub/docs/books/gutenberg/cdimages/pgdvd.iso --19:11:24-- ftp://ftp.ibiblio.org/pub/docs/books/gutenberg/cdimages/pgdvd.iso = `pgdvd.iso' Resolving ftp.ibiblio.org... 152.2.210.81 Connecting to ftp.ibiblio.org[152.2.210.81]:21... connected. Logging in as anonymous ... Logged in! == SYST ... done.== PWD ... done. == TYPE I ... done. == CWD /pub/docs/books/gutenberg/cdimages ... done. == SIZE pgdvd.iso ... done. == PORT ... done.== REST 2147483647 ... done. == RETR pgdvd.iso ... done. Length: 1,992,163,329 [-155,320,318 to go] (unauthoritative) 100%[+++] 2,147,483,647 --.--K/s File size limit exceeded (core dumped) I guess is a problem with the number representation in Linux (ie long int), perhaps is possible to use double type or a struct of int and make some special integer calculations within the structure, I really don't know , I never saw the source code, but I think is not a serious problem ... Well, best regards and Happy New Year! Miguel Angel Mendez
Crash on large files
You may already be aware (I havn't checked an up to date dev wget), but wget 1.8 seems to crash with the new progress bar when it has to handle a large file. Length: 495,585,280 (unauthoritative) 0% [ ] 0 --.--K/sETA --:--wget: progress.c:673: create_image: Assertion `p - bp-buffer = bp-width' failed. Aborted (core dumped) [snazbaz@mrpinky snaz]$ It works fine with smaller files. Anyway, I can stop it crashing with --progress=dot, so I assume this is a problem with the new bar type. Regards, Ian Packer [EMAIL PROTECTED]
