Re: Large file support
Am Mittwoch, den 23.02.2005, 23:13 +0100 schrieb Hrvoje Niksic: Hello, The most requested feature of the last several years finally arrives -- large file support. With this patch Wget should be able to download files larger than 2GB on systems that support them. dont know if its still interesting but there where no problems when building wget from cvs on different Linux architectures: http://experimental.ftbfs.de/build.php?arch=pkg=wget-cvs thx for including LFS.:) -- Nol Kthe noel debian.org Debian GNU/Linux, www.debian.org signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: Large file support
Noèl Köthe [EMAIL PROTECTED] writes: Am Mittwoch, den 23.02.2005, 23:13 +0100 schrieb Hrvoje Niksic: The most requested feature of the last several years finally arrives -- large file support. With this patch Wget should be able to download files larger than 2GB on systems that support them. dont know if its still interesting but there where no problems when building wget from cvs on different Linux architectures: http://experimental.ftbfs.de/build.php?arch=pkg=wget-cvs It is still interesting, thanks. Note, however, that all of the above architectures seem to run Debian GNU/Linux, which, being my current development platform, is pretty much guaranteed to build. That is, unless I did something stupid that would prevent compilation under a different CPU. thx for including LFS.:) It's been fun. And long overdue. :-)
RE: Large file support
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Maciej W. Rozycki Huh? It's you who should actually charge them for doing bug discovery for them. Yeah. Only please understand - I'm not a C programmer. I'm not an expert in Microsoft interaction (the fewer the better). I just have (legal!) access to a Visual Studio kit and use it, anything more in-depth unfortunately would require familiarizing far more with C in general and VS in particular than I have time available :( Anyway, with that level of vendor (non-)support, you should really consider switching to GCC, no matter how much hassle with libraries it involves. It's just an effort to be done once and *will* pay back. I know that, after all I could just download the free Borland compiler or MingW or for myself use the cygnus port (which I have installed anyway) and forget about VS. On the other hand I started compiling these binaries because I can and there seemed to be nobody else doing it and I feel it would be a shame dropping this compatibility exactly as it would be a shame dropping support for, say Ultrix. Just my (minimal) bit of support for a great tool - I do what I can but I won't become VSC proficient, time is never enough, family (and work) has precedence, I just can't afford it (although I'd really like to). Heiko -- -- PREVINET S.p.A. www.previnet.it -- Heiko Herold [EMAIL PROTECTED] [EMAIL PROTECTED] -- +39-041-5907073 ph -- +39-041-5907472 fax
RE: Large file support
On Mon, 28 Feb 2005, Herold Heiko wrote: Huh? It's you who should actually charge them for doing bug discovery for them. Yeah. Only please understand - I'm not a C programmer. I'm not an expert in Microsoft interaction (the fewer the better). I just have (legal!) access to a Visual Studio kit and use it, anything more in-depth unfortunately would require familiarizing far more with C in general and VS in particular than I have time available :( Well, with an ICE (internal compiler error), you've clearly found a compiler bug, no matter what you have passed to it. A compiler should accept anything (including /dev/random, speaking in *nix terms) and either accept it if valid input or reject it with a reasonable explanation (a parse error at worst). So if you just told them you you had received an ICE and dumped your preprocessed source causing the problem, they should be just happy about being able to fix yet another bug. You don't have to be a C programmer to do that. That's what I'd expect from a reasonable compiler vendor (or any software, actually). On the other hand I started compiling these binaries because I can and there seemed to be nobody else doing it and I feel it would be a shame dropping this compatibility exactly as it would be a shame dropping support for, say Ultrix. Just my (minimal) bit of support for a great tool - I do what I can but I won't become VSC proficient, time is never enough, family (and work) has precedence, I just can't afford it (although I'd really like to). Well, if you find your work useful and appreciated and you are satisfied with that, then by all means do continue it. It's just personally I wouldn't be so enthusiastic about working with a product that its vendor refuses to support. BTW, actually GCC dropped Ultrix support a while ago due to no interest and binutils support is about to go away, too. OTOH, you can still use older versions and for software like wget it doesn't really matter as long as you have a compiler toolchain available. Maciej
Re: Large file support
Hrvoje Niksic wrote: In other words, large files now work on Windows? I must admit, that was almost too easy. :-) Don't open the champagne bottle just yet :) Now could someone try this with Borland and/or Watcom and MingW? I'm pretty sure I broke them in some places, but it's near impossible to fix it without having the compilers available for testing. Patch attached. errno/WSAGetLastError() handling and reporting is still broken for all Win32 compilers. Search for SET_ERRNO() in the mail-archive. And also here: http://www.mail-archive.com/wget%40sunsite.dk/msg06475.html *_ERRNO() is in my working copy. Drop that until I come up with a use for them. --gv diff -u3 -Hb -r CVS-Latest/src/mswindows.h src/mswindows.h --- CVS-Latest/src/mswindows.h Fri Feb 25 23:23:21 2005 +++ src/mswindows.h Sat Feb 26 13:54:53 2005 @@ -83,7 +83,11 @@ /* Define a wgint type under Windows. */ typedef __int64 wgint; #define SIZEOF_WGINT 8 +#ifdef __GNUC__ +#define WGINT_MAX 9223372036854775807LL +#else #define WGINT_MAX 9223372036854775807I64 +#endif #define str_to_wgint str_to_int64 __int64 str_to_int64 (const char *, char **, int); @@ -99,7 +103,7 @@ # define fstat(fd, buf) _fstati64 (fd, buf) #endif -#if defined(_MSC_VER) +#if defined(_MSC_VER) || defined(__MINGW32__) # define struct_stat struct _stati64 #elif defined(__BORLANDC__) # define struct_stat struct stati64 diff -u3 -Hb -r CVS-Latest/src/sysdep.h src/sysdep.h --- CVS-Latest/src/sysdep.h Wed Feb 23 21:21:04 2005 +++ src/sysdep.hSat Feb 26 13:43:34 2005 @@ -108,6 +108,17 @@ #endif #endif +/* Hacks for setting/getting errno / h_errno. */ +#ifdef WINDOWS +# define GET_ERRNO() errno = WSAGetLastError() +# define SET_ERRNO(e) WSASetLastError (errno = (e)) +# define SET_H_ERRNO(e) WSASetLastError (e) +#else +# define GET_ERRNO() ((void)0) +# define SET_ERRNO(e) ((void)(errno = (e))) +# define SET_H_ERRNO(e) ((void)(h_errno = (e))) +#endif + /* Define a large integral type useful for storing large sizes that exceed sizes of one download, such as when printing the sum of all downloads. Note that this has nothing to do with large file @@ -127,7 +138,7 @@ typedef long long LARGE_INT; # define LARGE_INT_FMT %lld # else -# if _MSC_VER +# if defined(_MSC_VER) || defined(__MINGW32__) || defined(__WATCOMC__) /* Use __int64 under Windows. */ typedef __int64 LARGE_INT; # define LARGE_INT_FMT %I64
Re: Large file support
Gisle Vanem [EMAIL PROTECTED] writes: Hrvoje Niksic wrote: In other words, large files now work on Windows? I must admit, that was almost too easy. :-) Don't open the champagne bottle just yet :) Too late, the bottle is already empty. :-) Now could someone try this with Borland and/or Watcom and MingW? I'm pretty sure I broke them in some places, but it's near impossible to fix it without having the compilers available for testing. Patch attached. Thanks. Note that I fixed compilation under Borland in the meantime, because I found that the Borland command-line tools are available for free download. errno/WSAGetLastError() handling and reporting is still broken for all Win32 compilers. Search for SET_ERRNO() in the mail-archive. I remember that. At the time I disliked the idea of using the SET/GET_ERRNO pair instead of just using and assigning to errno. Another option was to simply set the (system) errno after the Winsock operations, and have our own strerror that recognizes them. (That assumes that Winsock errno values don't conflict with the system ones, which I believe is the case.)
Re: Large file support
Gisle Vanem [EMAIL PROTECTED] writes:
Another option was to simply set the (system) errno after the Winsock
operations, and have our own strerror that recognizes them. (That
assumes that Winsock errno values don't conflict with the system ones,
which I believe is the case.)
That assumption is wrong I think.
Winsock errors are in range 1 - 11004. Win32 API errors (from
WinErr.h) and libc errors (from errno.h) *do* overlap. E.g. both
EINTR and ERROR_TOO_MANY_OPEN_FILES both have the same value (4).
I'm not sure that we use the Win32 API calls and then check errno. If
we're doing that, we can set errno to GetLastError() + some large
constant specific to Wget after the calls.
We must therefore be careful using strerror() with errno codes only
and FormatMessage() with GetLastError (and WSAGetLastError). *Or*
cook up a smarter strerror() replacement.
I would lean toward a smarter strerror replacement. I'll definitely
try to get it working before the release of 1.10 so we finally get
decent error messages on Windows.
#undef strerror
char *win_strerror (int err)
{
static char buf[512];
char *p;
if (err = 0 err sys_nerr)
{
strncpy (buf, strerror(err), sizeof(buf)-1);
buf [sizeof(buf)-1] = '\0';
}
else
{
if (!get_winsock_error (err, buf, sizeof(buf))
!FormatMessage (FORMAT_MESSAGE_FROM_SYSTEM, NULL, err,
LANG_NEUTRAL, buf, sizeof(buf)-1, NULL))
sprintf (buf, Unknown error %d (%#x), err, err);
}
This is a good start -- but doesn't it suffer from the confusion
between EINTR and ERROR_TOO_MANY_OPEN_FILES?
RE: Large file support
From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Thursday, February 24, 2005 10:50 PM To: Maciej W. Rozycki Cc: Herold Heiko; wget@sunsite.dk Subject: Re: Large file support Maciej W. Rozycki [EMAIL PROTECTED] writes: Doesn't GCC work for this target? It does, in the form of Cygwin and MingW. But Heiko was using MS VC before, and we have catered to broken compilers before, so it doesn't hurt to try. Also, Cygwin requires a large installed environment. It may be possible to link statically and adapt everything in order to produce a single (large...) binary but that would be rather an ugly hack. MingW could be better, I really don't use it personally but I know the (standalone, portable) mame win32 binary is compiled with that. Nevertheless, Visual C++ is the de facto standard on the windows platforms, so supporting it if possible renders (self compiled) wget accessible to a larger audience. Now, if this means hacking the source too much I think the more recent vc++ only path shoule be tried first before throwing away vc++ support altogether. Said that, in retr.c simplifying the int rdsize line did not solve, but I tried the following, we have: #ifndef MIN # define MIN(i, j) ((i) = (j) ? (i) : (j)) #endif int rdsize = exact ? MIN (toread - sum_read, dlbufsize) : dlbufsize; double tmout = opt.read_timeout; Commenting out the double tmout... line removes the compiler error, OR exact ? (toread... (without MIN!) does compile, OR commenting out the #ifndef MIN..#endif does compile (in other words, MIN is already defined somewhere, how can I discover where?), however changing any occurence of MIN to XXXTEST still generates the compiler error, as Something simple like this int rdsize = 1; double tmout = opt.read_timeout; compiles, as does this int rdsize = dlbufsize; double tmout = opt.read_timeout; or this int rdsize = toread - sum_read; double tmout = opt.read_timeout; while this one int rdsize; rdsize = 1; double tmout = opt.read_timeout; fails with retr.c(263) : error C2143: syntax error : missing ';' before 'type' retr.c(269) : error C2065: 'tmout' : undeclared identifier (where line 263 is the double tmout=...). Any suggestions for other tests ? Heiko -- -- PREVINET S.p.A. www.previnet.it -- Heiko Herold [EMAIL PROTECTED] [EMAIL PROTECTED] -- +39-041-5907073 ph -- +39-041-5907472 fax
Re: Large file support
Herold Heiko [EMAIL PROTECTED] writes: Said that, in retr.c simplifying the int rdsize line did not solve, but I tried the following, we have: #ifndef MIN # define MIN(i, j) ((i) = (j) ? (i) : (j)) #endif int rdsize = exact ? MIN (toread - sum_read, dlbufsize) : dlbufsize; double tmout = opt.read_timeout; Commenting out the double tmout... line removes the compiler error, OR exact ? (toread... (without MIN!) does compile, OR commenting out the #ifndef MIN..#endif does compile (in other words, MIN is already defined somewhere, how can I discover where?), The whole point of #ifndef MIN ... #endif is to *use* the compiler-provided MIN where available. Dealing with a compiler crash is tricky, as it seems that every change has a potential of causing it to occur. Could you maybe tone down or remove the optimization flags when compiling retr.c?
RE: Large file support
On Fri, 25 Feb 2005, Herold Heiko wrote: Doesn't GCC work for this target? It does, in the form of Cygwin and MingW. But Heiko was using MS VC before, and we have catered to broken compilers before, so it doesn't hurt to try. Also, Cygwin requires a large installed environment. It may be possible to link statically and adapt everything in order to produce a single (large...) binary but that would be rather an ugly hack. MingW could be better, I really don't use it personally but I know the (standalone, portable) mame win32 binary is compiled with that. OK, thanks for the clarification. I'm not into non-*nix environments at all and with a *nix compiler you can normally drop in GCC as a replacement for the system C compiler without touching anything else, including the assembler, the linker and any libraries, be it shared or static (that's the point of having an ABI, after all). I'm not sure about other GCC languages, though -- they may require their own run-time libraries to be used instead of system ones; well, the Java compiler certainly does. ;-) Nevertheless, Visual C++ is the de facto standard on the windows platforms, so supporting it if possible renders (self compiled) wget accessible to a larger audience. Now, if this means hacking the source too much I think the more recent vc++ only path shoule be tried first before throwing away vc++ support altogether. Well, instead of scratching the head, how about filing a bug report? This way you should get a fix for the compiler and perhaps a description of a possible workaround for broken versions that we could use for wget. Commenting out the double tmout... line removes the compiler error, OR exact ? (toread... (without MIN!) does compile, OR commenting out the #ifndef MIN..#endif does compile (in other words, MIN is already defined somewhere, how can I discover where?), however changing any FYI, unless it's predefined, with GCC you'd use something like `gcc -E -dD foo.c' and search the output for the macro name. Maciej
RE: Large file support
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Maciej W. Rozycki Well, instead of scratching the head, how about filing a bug report? Ha :), would be nice. I suppose that would mean calling PSS, which (if things didn't change) means an immediate billing on your credit card (to be refunded later if there really was a problem). Anyway in this case the answer would probably be upgrade to VS.Net 2002. But see my other email... disabling optimization as Hrvoje suggested does work around the problem, and indeed I found some Microsoft articles suggesting exactly that. Heiko -- -- PREVINET S.p.A. www.previnet.it -- Heiko Herold [EMAIL PROTECTED] [EMAIL PROTECTED] -- +39-041-5907073 ph -- +39-041-5907472 fax
RE: Large file support
From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Friday, February 25, 2005 1:29 PM The whole point of #ifndef MIN ... #endif is to *use* the compiler-provided MIN where available. Dealing with a compiler crash is tricky, as it seems that every change has a potential of causing it to occur. Could you maybe tone down or remove the optimization flags when compiling retr.c? Does solve, in fact I found some MS articles suggesting the same thing. Attached patch does work around the problem by disabling optimization selectively.. I was able to retrieve a 2.5GB file with ftp. I tried to put a copy in the usual place (sunsite.dk by [EMAIL PROTECTED]), got autentication required - I suppose the password has changed. If still possible I'd like to put them there, for now a copy for general testing is at http://xoomer.virgilio.it/hherold/ Heiko -- -- PREVINET S.p.A. www.previnet.it -- Heiko Herold [EMAIL PROTECTED] [EMAIL PROTECTED] -- +39-041-5907073 ph -- +39-041-5907472 fax
RE: Large file support
On Fri, 25 Feb 2005, Herold Heiko wrote: Well, instead of scratching the head, how about filing a bug report? Ha :), would be nice. I suppose that would mean calling PSS, which (if things didn't change) means an immediate billing on your credit card (to be refunded later if there really was a problem). Huh? It's you who should actually charge them for doing bug discovery for them. Anyway in this case the answer would probably be upgrade to VS.Net 2002. I don't know how that actually relates to version 6 you've mentioned, but you should just obtain a bugfix release. But see my other email... disabling optimization as Hrvoje suggested does work around the problem, and indeed I found some Microsoft articles suggesting exactly that. Well, with the major version number of 6, you'd expect the product be already past its infancy, wouldn't you? Now I've heard some horror stories about C compilers crashing or producing bad code in early nineties (leading to many people avoiding any optimisation settings at all), but I've thought these times are over. Anyway, with that level of vendor (non-)support, you should really consider switching to GCC, no matter how much hassle with libraries it involves. It's just an effort to be done once and *will* pay back. Maciej
Re: Large file support
Herold Heiko [EMAIL PROTECTED] writes: Does solve, in fact I found some MS articles suggesting the same thing. Attached patch does work around the problem by disabling optimization selectively.. I was able to retrieve a 2.5GB file with ftp. In other words, large files now work on Windows? I must admit, that was almost too easy. :-) Now could someone try this with Borland and/or Watcom and MingW? I'm pretty sure I broke them in some places, but it's near impossible to fix it without having the compilers available for testing. I tried to put a copy in the usual place (sunsite.dk by [EMAIL PROTECTED]), got autentication required - I suppose the password has changed. If still possible I'd like to put them there, Mauro, are you aware of the password change?
RE: Large file support
Sorry, replied to wget-patches instead of redirecting it.
I tried a test compile just now, with Visual C++ 6 I get different errors:
string_t.[ch] - iswblank doesn't seem to be available, however there is a
int isspace( int c );
int iswspace( wint_t c );
Routine Required Header Compatibility
isspace ctype.h ANSI, Win 95, Win NT
iswspace ctype.h or wchar.h ANSI, Win 95, Win NT
isspace returns a non-zero value if c is a white-space character (0x09 -
0x0D or 0x20). iswspace returns a non-zero value if c is a wide character
that corresponds to a standard white-space character or is one of an
implementation-defined set of wide characters for which iswalnum is false.
Each of these routines returns 0 if c does not satisfy the test condition.
Also, the large file support patch introduced syntax errors in mswindows.h
and internal compiler error in retr.c and httpd.c:
Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
cd src
NMAKE
Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
cl /nologo /MT /O2 /I. /DWINDOWS /D_CONSOLE /DHAVE_CONFIG_H
/DHAVE_SSL /c http.c hash.c retr.c gen-md5.c gnu-md5.c
http.c
.\mswindows.h(109) : warning C4005: 'stat' : macro redefinition
.\mswindows.h(101) : see previous definition of 'stat'
http.c(503) : warning C4090: 'function' : different 'const' qualifiers
http.c(503) : warning C4022: 'xrealloc_real' : pointer mismatch for actual
parameter 1
http.c(521) : warning C4090: 'function' : different 'const' qualifiers
http.c(521) : warning C4022: 'xrealloc_real' : pointer mismatch for actual
parameter 1
http.c(675) : warning C4090: 'function' : different 'const' qualifiers
http.c(675) : warning C4022: 'xfree_real' : pointer mismatch for actual
parameter 1
http.c(1891) : warning C4133: 'function' : incompatible types - from 'struct
_stati64 *' to 'struct _stat *'
http.c(1900) : warning C4133: 'function' : incompatible types - from 'struct
_stati64 *' to 'struct _stat *'
http.c(1959) : warning C4133: 'function' : incompatible types - from 'struct
_stati64 *' to 'struct _stat *'
hash.c
.\mswindows.h(109) : warning C4005: 'stat' : macro redefinition
.\mswindows.h(101) : see previous definition of 'stat'
.\mswindows.h(121) : error C2143: syntax error : missing ')' before '*'
.\mswindows.h(121) : error C2143: syntax error : missing '{' before '*'
.\mswindows.h(121) : error C2059: syntax error : ')'
.\mswindows.h(121) : error C2059: syntax error : ';'
retr.c
.\mswindows.h(109) : warning C4005: 'stat' : macro redefinition
.\mswindows.h(101) : see previous definition of 'stat'
retr.c(920) : warning C4133: 'function' : incompatible types - from 'struct
_stati64 *' to 'struct _stat *'
gen-md5.c
.\mswindows.h(109) : warning C4005: 'stat' : macro redefinition
.\mswindows.h(101) : see previous definition of 'stat'
.\mswindows.h(121) : error C2143: syntax error : missing ')' before '*'
.\mswindows.h(121) : error C2143: syntax error : missing '{' before '*'
.\mswindows.h(121) : error C2059: syntax error : ')'
.\mswindows.h(121) : error C2059: syntax error : ';'
gnu-md5.c
.\mswindows.h(109) : warning C4005: 'stat' : macro redefinition
.\mswindows.h(101) : see previous definition of 'stat'
.\mswindows.h(121) : error C2143: syntax error : missing ')' before '*'
.\mswindows.h(121) : error C2143: syntax error : missing '{' before '*'
.\mswindows.h(121) : error C2059: syntax error : ')'
.\mswindows.h(121) : error C2059: syntax error : ';'
Generating Code...
retr.c(261) : fatal error C1001: INTERNAL COMPILER ERROR
(compiler file 'E:\8966\vc98\p2\src\P2\main.c', line 494)
Please choose the Technical Support command on the Visual C++
Help menu, or open the Technical Support help file for more information
http.c(1412) : warning C4761: integral size mismatch in argument; conversion
supplied
http.c(381) : fatal error C1001: INTERNAL COMPILER ERROR
(compiler file 'E:\8966\vc98\p2\src\P2\main.c', line 494)
Please choose the Technical Support command on the Visual C++
Help menu, or open the Technical Support help file for more information
NMAKE : fatal error U1077: 'cl' : return code '0x2'
Stop.
NMAKE : fatal error U1077: 'C:\PROGRA~1\MICROS~2\VC98\BIN\NMAKE.EXE' :
return code '0x2'
Stop.
Bye
Heiko
--
-- PREVINET S.p.A. www.previnet.it
-- Heiko Herold [EMAIL PROTECTED] [EMAIL PROTECTED]
-- +39-041-5907073 ph
-- +39-041-5907472 fax
-Original Message-
From: Hrvoje Niksic [mailto:[EMAIL PROTECTED]
Sent: Wednesday, February 23, 2005 11:14 PM
To: [EMAIL PROTECTED]
Subject: Large file support
The most requested feature of the last several years finally arrives
-- large file support. With this patch Wget should be able to
download files larger than 2GB on systems that support them.
ChangeLog:
2005-02-20 Hrvoje Niksic [EMAIL
Re: Large file support
Herold Heiko [EMAIL PROTECTED] writes:
I tried a test compile just now, with Visual C++ 6 I get different
errors:
Thanks for checking it.
string_t.[ch] - iswblank doesn't seem to be available,
For now, just remove string_t from the Makefile. It's not used
anywhere yet.
Also, the large file support patch introduced syntax errors in
mswindows.h and internal compiler error in retr.c and httpd.c:
I think I can fix the syntax errors; I'll need some guidance with the
internal compiler errors.
.\mswindows.h(109) : warning C4005: 'stat' : macro redefinition
.\mswindows.h(101) : see previous definition of 'stat'
OK, just remove the stat definition at line 109.
http.c(503) : warning C4090: 'function' : different 'const'
qualifiers
[...]
I don't quite understand these warnings. Did they occur before?
http.c(1891) : warning C4133: 'function' : incompatible types - from 'struct
_stati64 *' to 'struct _stat *'
This type of warning will disappear when you remove the bogus stat
definition.
.\mswindows.h(121) : error C2143: syntax error : missing ')' before '*'
.\mswindows.h(121) : error C2143: syntax error : missing '{' before '*'
.\mswindows.h(121) : error C2059: syntax error : ')'
.\mswindows.h(121) : error C2059: syntax error : ';'
This is a leftover from the Windows ftello implementation there used
to be at some point. I'll fix this in a patch.
Generating Code...
retr.c(261) : fatal error C1001: INTERNAL COMPILER ERROR
(compiler file 'E:\8966\vc98\p2\src\P2\main.c', line 494)
Please choose the Technical Support command on the Visual C++
Help menu, or open the Technical Support help file for more information
http.c(1412) : warning C4761: integral size mismatch in argument; conversion
supplied
http.c(381) : fatal error C1001: INTERNAL COMPILER ERROR
(compiler file 'E:\8966\vc98\p2\src\P2\main.c', line 494)
Please choose the Technical Support command on the Visual C++
Help menu, or open the Technical Support help file for more information
I don't quite know what to make of this. Any ideas?
RE: Large file support
From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Thursday, February 24, 2005 1:50 PM Thanks for checking it. Don't thank me, I'd like to do more but I have no time available :( http.c(503) : warning C4090: 'function' : different 'const' qualifiers [...] I don't quite understand these warnings. Did they occur before? Definitively, I trie with a rev from March 2004, same warnings. Generating Code... retr.c(261) : fatal error C1001: INTERNAL COMPILER ERROR (compiler file 'E:\8966\vc98\p2\src\P2\main.c', line 494) Please choose the Technical Support command on the Visual C++ Help menu, or open the Technical Support help file for more information http.c(1412) : warning C4761: integral size mismatch in argument; conversion supplied http.c(381) : fatal error C1001: INTERNAL COMPILER ERROR (compiler file 'E:\8966\vc98\p2\src\P2\main.c', line 494) Please choose the Technical Support command on the Visual C++ Help menu, or open the Technical Support help file for more information I don't quite know what to make of this. Any ideas? The problem was introduced with the large file patch. Using the http.c and retr.c from before that (with current headers and everything else) I get http.c(502) : warning C4090: 'function' : different 'const' qualifiers http.c(502) : warning C4022: 'xrealloc_real' : pointer mismatch for actual parameter 1 http.c(520) : warning C4090: 'function' : different 'const' qualifiers http.c(520) : warning C4022: 'xrealloc_real' : pointer mismatch for actual parameter 1 http.c(674) : warning C4090: 'function' : different 'const' qualifiers http.c(674) : warning C4022: 'xfree_real' : pointer mismatch for actual parameter 1 http.c(1733) : warning C4133: 'function' : incompatible types - from 'long *' to '__int64 *' http.c(1733) : warning C4133: 'function' : incompatible types - from 'long *' to '__int64 *' http.c(1874) : warning C4133: 'function' : incompatible types - from 'struct stat *' to 'struct _stati64 *' http.c(1883) : warning C4133: 'function' : incompatible types - from 'struct stat *' to 'struct _stati64 *' http.c(1942) : warning C4133: 'function' : incompatible types - from 'struct stat *' to 'struct _stati64 *' retr.c(197) : warning C4028: formal parameter 3 different from declaration retr.c(197) : warning C4028: formal parameter 4 different from declaration retr.c(197) : warning C4028: formal parameter 5 different from declaration retr.c(197) : warning C4028: formal parameter 6 different from declaration retr.c(493) : warning C4028: formal parameter 1 different from declaration retr.c(513) : warning C4028: formal parameter 1 different from declaration retr.c(918) : warning C4133: 'function' : incompatible types - from 'struct stat *' to 'struct _stati64 *' but it does compile. I tried to reverse stuff, as for retr.c by changing wgint back to long it compiles with these warnings: retr.c(199) : warning C4028: formal parameter 3 different from declaration retr.c(199) : warning C4028: formal parameter 4 different from declaration retr.c(199) : warning C4028: formal parameter 5 different from declaration retr.c(199) : warning C4028: formal parameter 6 different from declaration retr.c(495) : warning C4028: formal parameter 1 different from declaration retr.c(515) : warning C4028: formal parameter 1 different from declaration while http.c with long instead of wgint fails with http.c(1474) : error C2065: 'long_MAX' : undeclared identifier http.c(1750) : warning C4133: 'function' : incompatible types - from 'long *' to '__int64 *' http.c(1750) : warning C4133: 'function' : incompatible types - from 'long *' to '__int64 *' so the problem seems to be in there. Anything else I can try ? Heiko -- -- PREVINET S.p.A. www.previnet.it -- Heiko Herold [EMAIL PROTECTED] [EMAIL PROTECTED] -- +39-041-5907073 ph -- +39-041-5907472 fax
Re: Large file support
On Thu, 24 Feb 2005, Hrvoje Niksic wrote: Generating Code... retr.c(261) : fatal error C1001: INTERNAL COMPILER ERROR (compiler file 'E:\8966\vc98\p2\src\P2\main.c', line 494) Please choose the Technical Support command on the Visual C++ Help menu, or open the Technical Support help file for more information http.c(1412) : warning C4761: integral size mismatch in argument; conversion supplied http.c(381) : fatal error C1001: INTERNAL COMPILER ERROR (compiler file 'E:\8966\vc98\p2\src\P2\main.c', line 494) Please choose the Technical Support command on the Visual C++ Help menu, or open the Technical Support help file for more information I don't quite know what to make of this. Any ideas? Require a fixed version of the compiler? Doesn't GCC work for this target? Maciej
Re: Large file support
Herold Heiko [EMAIL PROTECTED] writes: http.c(503) : warning C4090: 'function' : different 'const' qualifiers [...] I don't quite understand these warnings. Did they occur before? Definitively, I trie with a rev from March 2004, same warnings. Then we can ignore them for now. I don't quite know what to make of this. Any ideas? The problem was introduced with the large file patch. Maybe the compiler doesn't handler complex __int64 expressions correctly. You could try to play with simplifying line 261 of retr.c (which it apparently fails on). For example, you could change this: int rdsize = exact ? MIN (toread - sum_read, dlbufsize) : dlbufsize; to something like: int rdsize; if (exact) rdsize = MIN (toread - sum_read, dlbufsize); else rdsize = dlbufsize; or even: int rdsize; if (exact) rdsize = (int) MIN (toread - sum_read, (wgint) dlbufsize); else rdsize = dlbufsize; I tried to reverse stuff, as for retr.c by changing wgint back to long That won't help because the routines in retr.c will get incorrect types in run-time.
Re: Large file support
Maciej W. Rozycki [EMAIL PROTECTED] writes: Doesn't GCC work for this target? It does, in the form of Cygwin and MingW. But Heiko was using MS VC before, and we have catered to broken compilers before, so it doesn't hurt to try.
Re: Large file support
On Sun, 20 Feb 2005, Hrvoje Niksic wrote: 1. I'd say that code like if ( sizeof(number) == 8 ) should have been a compile-time #ifdef rather than a run-time decision. Where do you see such code? grep 'if.*sizeof' *.c doesn't seem to show such examples. As I recall, it was in someone's suggested changes. Perhaps: http://software.lpetrov.net/wget-LFS/wget-LFS-20041113.patch Oh, OK. I thought you were referring to my LFS patch. Actually if (sizeof(number) == 8) is much more readable than any preprocessor clutter and yields exactly the same. Maciej
Re: Large file support
Maciej W. Rozycki [EMAIL PROTECTED] writes: Actually if (sizeof(number) == 8) is much more readable than any preprocessor clutter and yields exactly the same. Agreed, in some cases. In others it yields to pretty annoying compiler warnings.
Re: Large file support
From: Hrvoje Niksic [...] It is no small task to study Info-ZIP's source code. I did plan to look at it later, but at the time it was quicker to just ask. It's fairly easy to SEARCH [...]*.c, *.h LARGE_FILE_SUPPORT (or your local find/grep equivalent). Locating zip_fzofft would also be pretty easy. Anyway, why the unfriendliness? [...] Not unfriendly, just a bit frustrated. Steven M. Schweda (+1) 651-699-9818 382 South Warwick Street[EMAIL PROTECTED] Saint Paul MN 55105-2547
Re: Large file support
On Mon, 21 Feb 2005, Hrvoje Niksic wrote: Actually if (sizeof(number) == 8) is much more readable than any preprocessor clutter and yields exactly the same. Agreed, in some cases. In others it yields to pretty annoying compiler warnings. What kind of warnings? It's valid C. Maciej
Re: Large file support
Maciej W. Rozycki [EMAIL PROTECTED] writes: On Mon, 21 Feb 2005, Hrvoje Niksic wrote: Actually if (sizeof(number) == 8) is much more readable than any preprocessor clutter and yields exactly the same. Agreed, in some cases. In others it yields to pretty annoying compiler warnings. What kind of warnings? Comparing constants. Unreachable code. It's valid C. Never said it wasn't -- those are warnings, not errors.
Re: Large file support
Maciej W. Rozycki [EMAIL PROTECTED] writes: Besides, despite sizeof(foo) being a constant, you can't move a comparison against it to cpp. You can, Autoconf allows you to check for size of foo, which gives you a SIZEOF_FOO preprocessor constant. Then you can write things like: #if SIZEOF_FOO == 4 /* Assume 32-bit foo */ #else /* Don't make that assumption */ #endif
Re: Large file support
On Mon, 21 Feb 2005, Hrvoje Niksic wrote: Besides, despite sizeof(foo) being a constant, you can't move a comparison against it to cpp. You can, Autoconf allows you to check for size of foo, which gives you a SIZEOF_FOO preprocessor constant. Then you can write things like: #if SIZEOF_FOO == 4 /* Assume 32-bit foo */ #else /* Don't make that assumption */ #endif This may work as a workaround for this particular case and may often be a reasonable choice regardless of the compiler's shortcoming. But for some code it may be highly troublesome to maintain, especially if you get sizeof(foo) at both sides of an (in)equation. It's hardly a gain to give up code readability to satisfy a picky compiler. Maciej
Re: Large file support
[EMAIL PROTECTED] (Steven M. Schweda) writes: 1. I'd say that code like if ( sizeof(number) == 8 ) should have been a compile-time #ifdef rather than a run-time decision. Where do you see such code? grep 'if.*sizeof' *.c doesn't seem to show such examples. As I recall, it was in someone's suggested changes. Perhaps: http://software.lpetrov.net/wget-LFS/wget-LFS-20041113.patch Oh, OK. I thought you were referring to my LFS patch. The Info-ZIP code uses one function with a ring of string buffers to ease the load on the programmer. That makes sense. I assume the print function also receives an integer argument specifying the ring position? Why assume? Why not look? Sorry about that. It is no small task to study Info-ZIP's source code. I did plan to look at it later, but at the time it was quicker to just ask. You're right, that's much better. I've now written a new version that does away with the _second function. The caller still has to be aware of the ring size limitation, but that's hard to avoid without making the interface hairier. Say, is that wheel your own invention? It was proposed by Roman Bednarek in [EMAIL PROTECTED]. Anyway, why the unfriendliness? Again, sorry about not having had the time to study a large program like Info-ZIP. The LFS patch I'm working on is still in its infancy, and there is a lot of room for improvements.
Re: Large file support (was Re: Back after a while)
Dave Yeo [EMAIL PROTECTED] writes: ps anyone getting a bunch of what look like viruses on the wget-patches list? I just noticed them on gmane. I've now asked the SunSITE.dk staff to deploy the kind of virus/spam protection currently used by this list (confirmation required for non-subscribers to post), which seems to work well.
Re: Large file support
[EMAIL PROTECTED] (Steven M. Schweda) writes: 1. I'd say that code like if ( sizeof(number) == 8 ) should have been a compile-time #ifdef rather than a run-time decision. Where do you see such code? grep 'if.*sizeof' *.c doesn't seem to show such examples. 2. Multiple functions like print_number_as_string() and print_second_number_as_string() (and so on?) look like a real pain to use. They're just a hack to get multiple %s's to work. If you have a better idea of how to code them, feel free to suggest! The Info-ZIP code uses one function with a ring of string buffers to ease the load on the programmer. That makes sense. I assume the print function also receives an integer argument specifying the ring position? 3. print_second_number_as_string()? Are you sure the names are long enough? (VMS C (by default) truncates externals longer than 31 characters, so I worry about these things even if no one else objects to typing for days on one statement.) That identifier is 29 characters long, so it should work AFAIK. If someone knows of a system (still in use) that doesn't support it, let us know.
Re: Large file support
Roman Bednarek [EMAIL PROTECTED] writes: The Info-ZIP code uses one function with a ring of string buffers to ease the load on the programmer. That makes sense. I assume the print function also receives an integer argument specifying the ring position? The function can have a circular buffer with current free slot index, so the integer argument will not be necessary, and everything will work as long as the buffer is long enough to support worst case. You're right, that's much better. I've now written a new version that does away with the _second function. The caller still has to be aware of the ring size limitation, but that's hard to avoid without making the interface hairier.
Large file support (was: Back after a while)
Leonid [EMAIL PROTECTED] writes: Steven and Hrvoje, wget-1.9.1 has a function number_to_string which is in fact a home-made equivalent to printf () %ld. Yes, but that function is merely an optimization used to avoid frequent calls to sprintf(buf, %ld, num). Wget does in fact in many places use %ld, which is revealed when you grep for %ld in the source code. We could replace %ld with %s and a call to something like number_to_string, but which returned a static buffer. Of course, this would break format strings that print two numbers, so we'd need a way to specify a different buffer as well. The resulting code would be a bit clunky, but would work well and remain portable, I think. In the LFS patch which I submitted I made a sister-routine number_to_string_64 which is equivalent to printf() %llu. Thus, wget does not rely on %ll or %llu. Is your LFS patch available on the web? I was unsubscribed from wget-patches due to bounces from my address and there is no web archive (that I know of) of that list.
Another crack at large file support
Here is my attempt at adding large file support to Wget in a manner I
think is fairly portable. I'd be interested to know if this breaks
compilation for anyone.
The patch is purely experimental; I haven't even written the ChangeLog
entries yet. After applying it, don't forget to run autoheaders and
autoconf and to re-run configure.
The patch basically replaces all uses of `long' with `wgint', a type
that is defined as (LFS version of) off_t, except under Windows, where
it is simply `long long'. Systems with LFS and Windows will have
64-bit wgint, whereas others will have a 32-bit type.
number_to_string has been modified to accept a wgint argument, and a
new function print_number_as_string has been added to facilitate
printing of wgints.
Index: configure.in
===
RCS file: /pack/anoncvs/wget/configure.in,v
retrieving revision 1.73
diff -u -r1.73 configure.in
--- configure.in2003/11/26 22:46:13 1.73
+++ configure.in2005/02/18 23:25:36
@@ -183,6 +183,14 @@
AC_CHECK_SIZEOF(long long)
dnl
+dnl Check for large file support. This check needs to come fairly
+dnl early because it could (in principle) affect whether functions and
+dnl headers are available, whether they work, etc.
+dnl
+AC_SYS_LARGEFILE
+AC_CHECK_SIZEOF(off_t)
+
+dnl
dnl Checks for non-universal or system-specific types.
dnl
AC_TYPE_SIZE_T
Index: src/ftp-basic.c
===
RCS file: /pack/anoncvs/wget/src/ftp-basic.c,v
retrieving revision 1.36
diff -u -r1.36 ftp-basic.c
--- src/ftp-basic.c 2003/11/21 01:48:05 1.36
+++ src/ftp-basic.c 2005/02/18 23:25:37
@@ -897,15 +897,13 @@
/* Sends REST command to the FTP server. */
uerr_t
-ftp_rest (int csock, long offset)
+ftp_rest (int csock, wgint offset)
{
char *request, *respline;
int nwritten;
uerr_t err;
- static char numbuf[24]; /* Buffer for the number */
- number_to_string (numbuf, offset);
- request = ftp_request (REST, numbuf);
+ request = ftp_request (REST, print_number_as_string (offset));
nwritten = fd_write (csock, request, strlen (request), -1);
if (nwritten 0)
{
@@ -1114,7 +1112,7 @@
/* Sends the SIZE command to the server, and returns the value in 'size'.
* If an error occurs, size is set to zero. */
uerr_t
-ftp_size (int csock, const char *file, long int *size)
+ftp_size (int csock, const char *file, wgint *size)
{
char *request, *respline;
int nwritten;
@@ -1150,7 +1148,7 @@
}
errno = 0;
- *size = strtol (respline + 4, NULL, 0);
+ *size = str_to_wgint (respline + 4, NULL, 0);
if (errno)
{
/*
Index: src/ftp-ls.c
===
RCS file: /pack/anoncvs/wget/src/ftp-ls.c,v
retrieving revision 1.28
diff -u -r1.28 ftp-ls.c
--- src/ftp-ls.c2004/12/09 01:20:39 1.28
+++ src/ftp-ls.c2005/02/18 23:25:39
@@ -213,7 +213,7 @@
if (i != 12)
{
char *t = tok - 2;
- long mul = 1;
+ wgint mul = 1;
for (cur.size = 0; t line ISDIGIT (*t); mul *= 10, t--)
cur.size += mul * (*t - '0');
@@ -519,7 +519,7 @@
cur.type = FT_PLAINFILE;
cur.size = atoi(tok);
cur.perms = 0644;
- DEBUGP((File, size %ld bytes\n, cur.size));
+ DEBUGP((File, size %s bytes\n, print_number_as_string (cur.size)));
}
cur.linkto = NULL;
Index: src/ftp.c
===
RCS file: /pack/anoncvs/wget/src/ftp.c,v
retrieving revision 1.88
diff -u -r1.88 ftp.c
--- src/ftp.c 2004/02/25 23:45:24 1.88
+++ src/ftp.c 2005/02/18 23:25:41
@@ -83,12 +83,12 @@
/* Look for regexp ( *[0-9]+ *byte (literal parenthesis) anywhere in
- the string S, and return the number converted to long, if found, 0
+ the string S, and return the number converted to wgint, if found, 0
otherwise. */
-static long
+static wgint
ftp_expected_bytes (const char *s)
{
- long res;
+ wgint res;
while (1)
{
@@ -234,7 +234,7 @@
connection to the server. It always closes the data connection,
and closes the control connection in case of error. */
static uerr_t
-getftp (struct url *u, long *len, long restval, ccon *con)
+getftp (struct url *u, wgint *len, wgint restval, ccon *con)
{
int csock, dtsock, local_sock, res;
uerr_t err;
@@ -243,10 +243,10 @@
char *tms, *tmrate;
int cmd = con-cmd;
int pasv_mode_open = 0;
- long expected_bytes = 0L;
+ wgint expected_bytes = 0L;
int rest_failed = 0;
int flags;
- long rd_size;
+ wgint rd_size;
assert (con != NULL);
assert (con-target != NULL);
@@ -770,7 +770,7 @@
if (restval (cmd DO_RETR))
{
if (!opt.server_response)
- logprintf (LOG_VERBOSE, == REST %ld ... , restval);
+ logprintf (LOG_VERBOSE, == REST %s
Large file support (was: Back after a while)
Hrvoje, Is your LFS patch available on the web? I was unsubscribed from wget-patches due to bounces from my address and there is no web archive (that I know of) of that list. Yes. It is in http://software.lpetrov.net/wget-LFS/ Another bonus of using routines number_to_string and legible is that you can put separators in big numbers. For example, when I see the number 12392943028, it is difficult to say from the first glance whether wget has downloaded 12 Gb or 1.2 Gb. The number 12,392,943,028 seems more legible. Hmmm. I've already seen a request to support localization of this separator. This is for the future... We could replace %ld with %s and a call to something like number_to_string, but which returned a static buffer. Oh, sure. Leonid
Large file support (was: Back after a while)
It's not my program (obviously), but: 1. I'd say that code like if ( sizeof(number) == 8 ) should have been a compile-time #ifdef rather than a run-time decision. 2. Multiple functions like print_number_as_string() and print_second_number_as_string() (and so on?) look like a real pain to use. The Info-ZIP code uses one function with a ring of string buffers to ease the load on the programmer. So long as you don't put too many calls into the same printf(), it's pretty painless. 3. print_second_number_as_string()? Are you sure the names are long enough? (VMS C (by default) truncates externals longer than 31 characters, so I worry about these things even if no one else objects to typing for days on one statement.) Steven M. Schweda (+1) 651-699-9818 382 South Warwick Street[EMAIL PROTECTED] Saint Paul MN 55105-2547
Large file support (was Re: Back after a while)
On Wed, 16 Feb 2005 00:52:56 -0600 (CST), Steven M. Schweda wrote: I don't claim that it's a perfect scheme, but it would probably pay to look at the beta code for Zip 3.0 and/or UnZip 6.0 to see how these things are done in a _very_ portable program. (Before doing something even more lame.) Is the source for Zip 3.0/UnZip 6.0 publicaly available? Also another project to look at for large file support is cdrecord (also star by the same author)(http://cdrecord.berlios.de/old/private/cdrecord.html) which includes tests in aclocal.m4 and configure. Dave ps anyone getting a bunch of what look like viruses on the wget-patches list?
Re: Large file support (was Re: Back after a while)
Is the source for Zip 3.0/UnZip 6.0 publicaly available? I believe that some relatively recent beta code (Zip 3.0d, UnZip 6.0b) is available under: ftp://ftp.info-zip.org/pub/infozip/OLD/beta/ and probably various mirrors around the world. You might wish to start at http://www.info-zip.org/Zip.html#Sources, choose a host, then look for the new stuff there. Zip 3.0e and UnZip 6.0c (still beta) are in the works (getting closer, but with no firm date), but I believe that the large-file code had largely settled down in what you can find now. Steven M. Schweda (+1) 651-699-9818 382 South Warwick Street[EMAIL PROTECTED] Saint Paul MN 55105-2547
large file support
http://cvs.sunsite.dk/viewcvs.cgi/wget/NEWS?rev=WGET_1_9_1content-type=text/plain I was using wget on freebsd to mirror a site with an ISO image of 3.3GB, and noticed that it went berserk when I re-ran the wget, and continued to download and extend the file! I checked the news above and it didn't seem to mention large file support. However, I found this page: http://www.linuxsecurity.com/content/view/106766/102/ which suggested that someone at Redhat has added large file support, but I guess they didn't submit the patch to you. Paul
Large File Support, and broken CVS on Mac OS X
Greg, A short clarification: My patch http://software.lpetrov.net/wget-LFS/ is a derivative of Alvaro Lopez Ortega patch http://www.alobbs.com/modules.php?op=modloadname=Newsfile=articlesid=380mode=threadorder=0thold=0 I took his patch as a basis and corrected remaining bugs. Leonid
Large File Support, and broken CVS on Mac OS X
El 10/01/2005, a las 5:07, Mark Wiebe escribió: This occurred after many hours of downloading. At 2GB, the bytes downloaded started reporting negative numbers, but it still appeared to be working. This is the third post to this list in three days about (lack of) large-file support in wget. The issue is addressed in the FAQ: http://www.gnu.org/software/wget/faq.html#3.1 A lot of people obviously don't read the FAQ, though. And unfortunately, there's no easy way to search the mailing list archives before posting. If there were, I suspect that searches for things like DVD, GB, and large file would turn up hundreds of hits. (Yes, there are no doubt some web-based archives of the mailing list, such as http://www.mail-archive.com/wget%40sunsite.dk/, but there's no single official archive linked to from the official wget page or in the wget documentation.) It's great that some people have already come up with platform-specific patches to address the shortcoming. Unfortunately, it seems that it's quite a bit harder to do it in a cross-platform manner, hence the delay in releasing an update. I have zero knowledge of other platforms, but I do at least have a Mac OS X box in front of me so I am going to post some comments about these patches on Mac OS X. I suspect there might be some commonality between Mac OS X and the other BSD-based operating systems on which wget runs. [Yes, yes, curl has support for large files. But the version of curl that comes with Mac OS X (even the latest released version, 10.3.7) is old (7.10.2) and therefore doesn't have large file support. I have tested the latest curl release (built from source) with large files and it seems to work without any config tweaks.] I know all these patches were most likely written for and tested on Linux, but I thought I'd give it a try anyway. I wanted to try all three patches that I've see so far (the one by Leonid Petrov available from http://software.lpetrov.net/wget-LFS/; the one by Alvaro Lopez Ortega from http://www.alobbs.com/modules.php? op=modloadname=Newsfile=articlesid=380mode=threadorder=0thold=0; and the one by Dmitry Antipov which I didn't find for download anywhere but which was attached to a post to this list) but the current CVS version of wget is broken on Mac OS X. Don't know if it is on any other platforms. Specifically, the build fails during the make in the following way: cd . autoheader echo timestamp ./stamp-h.in cd src make CC='gcc' CPPFLAGS='' DEFS='-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\' CFLAGS='-O2 -Wall -Wno-implicit' LDFLAGS='' LIBS='-lssl -lcrypto -ldl ' prefix='/usr/local' exec_prefix='/usr/local' bindir='/usr/local/bin' infodir='/usr/local/info' mandir='/usr/local/man' manext='1' /Users/ghurrell/tmp/dmitry/wget/src gcc -I. -I.-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\ -O2 -Wall -Wno-implicit -c cmpt.c In file included from cmpt.c:35: string.h:35: error: parse error before PARAMS make[1]: *** [cmpt.o] Error 1 make: *** [src] Error 2 I don't know why, but some kind of change has been made which causes gcc to use wget's src/string.h file instead of the system string.h file (at /usr/include/string.h). As a quick diagnostic kludge, I deleted the wget src/string.h file and confirmed that the build then completed without errors. The resulting built wget runs, but I didn't have any confidence in it so I didn't test it nor was I going to apply these patches to a broken CVS tree. The last time I pulled the source from CVS and built it on Mac OS X without problems was 21 November 2004. In any case, even though I couldn't test those patches on Mac OS X, I did take a good look at them (with my novice's eye!) Here are some notes, for what it's worth: 1. Mac OS X does have strtoll. 2. It also has off_t type, which is defined as a 64 bit signed integer (int64_t, which in turn is defined as long long). 3. The printf family of functions has support for format strings like %llu. 4. There is one potential problem that I can see with the patches by Alvaro and Leonid when applied to Mac OS X. In utils.c there is a call to ftell, expecting an off_t return value, but the function prototype declares that it returns a long return value: long ftell(FILE *stream); There is another function, ftello, which does what you want: off_t ftello(FILE *stream); 5. The only other problem I could see was in the patch by Dmitry, where he passes O_LARGEFILE as an option to open, but that option isn't documented on Mac OS X. And that's where my technical knowledge runs out! Hope the information is useful to people who are more knowledgeable than me, but might not have access to a Mac OS X box! Greg
Re: Large File Support, and broken CVS on Mac OS X
Alle 16:28, lunedì 10 gennaio 2005, hai scritto: El 10/01/2005, a las 5:07, Mark Wiebe escribió: This occurred after many hours of downloading. At 2GB, the bytes downloaded started reporting negative numbers, but it still appeared to be working. This is the third post to this list in three days about (lack of) large-file support in wget. The issue is addressed in the FAQ: http://www.gnu.org/software/wget/faq.html#3.1 A lot of people obviously don't read the FAQ, though. And unfortunately, there's no easy way to search the mailing list archives before posting. If there were, I suspect that searches for things like DVD, GB, and large file would turn up hundreds of hits. hi greg, i am very glad you have brought up this problem, since i also think the time has come to start working seriously on LFS support for wget. (Yes, there are no doubt some web-based archives of the mailing list, such as http://www.mail-archive.com/wget%40sunsite.dk/, but there's no single official archive linked to from the official wget page or in the wget documentation.) well, the official wget page: http://www.gnu.org/software/wget reports the following URLs: http://fly.cc.fer.hr/archive/wget http://www.mail-archive.com/wget%40sunsite.dk/ http://www.geocrawler.com/archives/3/409/ as mailing list archives. perhaps you're suggesting we should choose one of them as the official one? It's great that some people have already come up with platform-specific patches to address the shortcoming. Unfortunately, it seems that it's quite a bit harder to do it in a cross-platform manner, hence the delay in releasing an update. I have zero knowledge of other platforms, but I do at least have a Mac OS X box in front of me so I am going to post some comments about these patches on Mac OS X. I suspect there might be some commonality between Mac OS X and the other BSD-based operating systems on which wget runs. i completely agree. I know all these patches were most likely written for and tested on Linux, but I thought I'd give it a try anyway. I wanted to try all three patches that I've see so far (the one by Leonid Petrov available from http://software.lpetrov.net/wget-LFS/; the one by Alvaro Lopez Ortega from http://www.alobbs.com/modules.php? op=modloadname=Newsfile=articlesid=380mode=threadorder=0thold=0; and the one by Dmitry Antipov which I didn't find for download anywhere but which was attached to a post to this list) but the current CVS version of wget is broken on Mac OS X. Don't know if it is on any other platforms. Specifically, the build fails during the make in the following way: cd . autoheader echo timestamp ./stamp-h.in cd src make CC='gcc' CPPFLAGS='' DEFS='-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\' CFLAGS='-O2 -Wall -Wno-implicit' LDFLAGS='' LIBS='-lssl -lcrypto -ldl ' prefix='/usr/local' exec_prefix='/usr/local' bindir='/usr/local/bin' infodir='/usr/local/info' mandir='/usr/local/man' manext='1' /Users/ghurrell/tmp/dmitry/wget/src gcc -I. -I.-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\ -O2 -Wall -Wno-implicit -c cmpt.c In file included from cmpt.c:35: string.h:35: error: parse error before PARAMS make[1]: *** [cmpt.o] Error 1 make: *** [src] Error 2 I don't know why, but some kind of change has been made which causes gcc to use wget's src/string.h file instead of the system string.h file (at /usr/include/string.h). As a quick diagnostic kludge, I deleted the wget src/string.h file and confirmed that the build then completed without errors. The resulting built wget runs, but I didn't have any confidence in it so I didn't test it nor was I going to apply these patches to a broken CVS tree. The last time I pulled the source from CVS and built it on Mac OS X without problems was 21 November 2004. this is my fault. i tested the code under linux and it worked well. i didn't have any other platform to test it so i did what developers are supposed to do in this case: just commit the code and wait for someone like you to send a bug report ;-) i'll fix this problem ASAP. anyway, i suddenly realize string.h is probably a very poor choice for a filename to be included in wget. In any case, even though I couldn't test those patches on Mac OS X, I did take a good look at them (with my novice's eye!) Here are some notes, for what it's worth: 1. Mac OS X does have strtoll. 2. It also has off_t type, which is defined as a 64 bit signed integer (int64_t, which in turn is defined as long long). 3. The printf family of functions has support for format strings like %llu. 4. There is one potential problem that I can see with the patches by Alvaro and Leonid when applied to Mac OS X. In utils.c there is a call to ftell, expecting an off_t return value, but the function prototype declares that it returns a long return value: long ftell
