El 10/01/2005, a las 5:07, Mark Wiebe escribió:
This occurred after many hours of downloading. At 2GB, the bytes downloaded started reporting negative numbers, but it still appeared to be working.
This is the third post to this list in three days about (lack of) large-file support in wget. The issue is addressed in the FAQ:
http://www.gnu.org/software/wget/faq.html#3.1
A lot of people obviously don't read the FAQ, though. And unfortunately, there's no easy way to search the mailing list archives before posting. If there were, I suspect that searches for things like "DVD", "GB", and "large file" would turn up hundreds of hits. (Yes, there are no doubt some web-based archives of the mailing list, such as <http://www.mail-archive.com/wget%40sunsite.dk/>, but there's no single official archive linked to from the official wget page or in the wget documentation.)
It's great that some people have already come up with platform-specific patches to address the shortcoming. Unfortunately, it seems that it's quite a bit harder to do it in a cross-platform manner, hence the delay in releasing an update. I have zero knowledge of other platforms, but I do at least have a Mac OS X box in front of me so I am going to post some comments about these patches on Mac OS X. I suspect there might be some commonality between Mac OS X and the other BSD-based operating systems on which wget runs.
[Yes, yes, curl has support for large files. But the version of curl that comes with Mac OS X (even the latest released version, 10.3.7) is old (7.10.2) and therefore doesn't have large file support. I have tested the latest curl release (built from source) with large files and it seems to work without any config tweaks.]
I know all these patches were most likely written for and tested on Linux, but I thought I'd give it a try anyway. I wanted to try all three patches that I've see so far (the one by Leonid Petrov available from <http://software.lpetrov.net/wget-LFS/>; the one by Alvaro Lopez Ortega from <http://www.alobbs.com/modules.php? op=modload&name=News&file=article&sid=380&mode=thread&order=0&thold=0>; and the one by Dmitry Antipov which I didn't find for download anywhere but which was attached to a post to this list) but the current CVS version of wget is broken on Mac OS X. Don't know if it is on any other platforms. Specifically, the build fails during the "make" in the following way:
cd . && autoheader
echo timestamp > ./stamp-h.in
cd src && make CC='gcc' CPPFLAGS='' DEFS='-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\"/usr/local/etc/wgetrc\" -DLOCALEDIR=\"/usr/local/share/locale\"' CFLAGS='-O2 -Wall -Wno-implicit' LDFLAGS='' LIBS='-lssl -lcrypto -ldl ' prefix='/usr/local' exec_prefix='/usr/local' bindir='/usr/local/bin' infodir='/usr/local/info' mandir='/usr/local/man' manext='1'
/Users/ghurrell/tmp/dmitry/wget/src
gcc -I. -I. -DHAVE_CONFIG_H -DSYSTEM_WGETRC=\"/usr/local/etc/wgetrc\" -DLOCALEDIR=\"/usr/local/share/locale\" -O2 -Wall -Wno-implicit -c cmpt.c
In file included from cmpt.c:35:
string.h:35: error: parse error before "PARAMS"
make[1]: *** [cmpt.o] Error 1
make: *** [src] Error 2
I don't know why, but some kind of change has been made which causes gcc to use wget's "src/string.h" file instead of the system string.h file (at /usr/include/string.h). As a quick diagnostic kludge, I deleted the wget src/string.h file and confirmed that the build then completed without errors. The resulting built wget runs, but I didn't have any confidence in it so I didn't test it nor was I going to apply these patches to a broken CVS tree. The last time I pulled the source from CVS and built it on Mac OS X without problems was 21 November 2004.
In any case, even though I couldn't test those patches on Mac OS X, I did take a good look at them (with my novice's eye!) Here are some notes, for what it's worth:
1. Mac OS X does have strtoll.
2. It also has "off_t" type, which is defined as a 64 bit signed integer ("int64_t", which in turn is defined as long long).
3. The printf family of functions has support for format strings like "%llu".
4. There is one potential problem that I can see with the patches by Alvaro and Leonid when applied to Mac OS X. In utils.c there is a call to "ftell", expecting an off_t return value, but the function prototype declares that it returns a long return value:
long ftell(FILE *stream);
There is another function, ftello, which does what you want:
off_t ftello(FILE *stream);
5. The only other problem I could see was in the patch by Dmitry, where he passes "O_LARGEFILE" as an option to open, but that option isn't documented on Mac OS X.
And that's where my technical knowledge runs out! Hope the information is useful to people who are more knowledgeable than me, but might not have access to a Mac OS X box!
Greg
