Re: Make -p work with framed pages.
On 1 Dec 2001 at 4:04, Hrvoje Niksic wrote: As a TODO entry summed up: * -p should probably go _two_ more hops on FRAMESET pages. More generally, I think it probably needs to be made to work for nested framesets too.
Re: Wget 1.8-beta1 now available
On 01/12/2001 19:44:44 John Poltorak wrote: On Sat, Dec 01, 2001 at 04:30:47PM +0100, Hrvoje Niksic wrote: John Poltorak [EMAIL PROTECTED] writes: Is it possible to include OBJEXT in Makefile.in to make this more cross-platform? I suppose so. I mean, o is already defined to .@U@o, but I'm not exactly sure what the U is supposed to stand for. It's looks to me as though @U@ is set up for some variable substitution, but I can't work out what for... Maybe it's getting replaced by NULL. I know next to nothing about how Auto* is (supposed to be) working, but I've seen lots of sed commands in If @U@ is doing a variable substitution, then it'll expand to something _before_ o (if @U@ - bar, then this will result in a dependency involving .baro) (looking through configure) Wget's configure contains this towards the end: s%@U@%$U%g U seems to be related to ansi2knr: if(can use prototypes) U= ANSI2KNR= else U=_ ANSI2KNR=./ansi2knr endif This will result in dependencies written as ._o if ansi2knr was run over the sources. This forces me to conclude that using @U@ _CAN_NOT_ and _WILL_NOT_ change .o to .obj I think .@U@o might need to be replaced with .@U@@objext@ (if there is such a beast, in analogy with @exeext@) Csaba -- Csaba Ráduly, Software Engineer Sophos Anti-Virus email: [EMAIL PROTECTED]http://www.sophos.com US Support: +1 888 SOPHOS 9 UK Support: +44 1235 559933
Win ssl bug
The current CVS 1.8 needs this in order to compile with ssl: diff -urbB wget/windows/Makefile.src wget.wip/windows/Makefile.src --- wget/windows/Makefile.src Thu Nov 29 14:30:24 2001 +++ wget.wip/windows/Makefile.src Mon Dec 3 13:39:58 2001 @@ -55,12 +55,12 @@ SRC = cmpt.c safe-ctype.c connect.c host.c http.c netrc.c ftp-basic.c ftp.c \ ftp-ls.c ftp-opie.c getopt.c hash.c headers.c html-parse.c html-url.c \ progress.c retr.c recur.c res.c url.c cookies.c init.c utils.c main.c \ - version.c mswindows.c fnmatch.c gnu-md5.c rbuf.c log.c $(SSLSRC) + version.c mswindows.c fnmatch.c gen-md5.c gnu-md5.c rbuf.c log.c $(SSLSRC) OBJ = cmpt$o safe-ctype$o connect$o host$o http$o netrc$o ftp-basic$o ftp$o \ ftp-ls$o ftp-opie$o getopt$o hash$o headers$o html-parse$o html-url$o \ progress$o retr$o recur$o res$o url$o cookies$o init$o utils$o main$o \ - version$o mswindows$o fnmatch$o gnu-md5$o rbuf$o log$o $(SSLOBJ) + version$o mswindows$o fnmatch$o gen-md5$o gnu-md5$o rbuf$o log$o $(SSLOBJ) .SUFFIXES: .c .obj diff -urbB wget/windows/Makefile.src.bor wget.wip/windows/Makefile.src.bor --- wget/windows/Makefile.src.bor Thu Nov 29 14:30:24 2001 +++ wget.wip/windows/Makefile.src.bor Mon Dec 3 10:03:37 2001 @@ -39,6 +39,8 @@ html.obj+ host.obj+ headers.obj+ +gen-md5.obj+ +gnu-md5.obj+ getopt.obj+ ftp-opie.obj+ ftp-ls.obj+ However it still won' work due to mising gen_sslfunc.obj : error LNK2001: unresolved external symbol _lrand48 gen_sslfunc.obj : error LNK2001: unresolved external symbol _srand48 In fact looking through the VC++ documentation there doesn't seem to be a rand48() available. Heiko -- -- PREVINET S.p.A.[EMAIL PROTECTED] -- Via Ferretto, 1ph x39-041-5907073 -- I-31021 Mogliano V.to (TV) fax x39-041-5907087 -- ITALY
Re: Win ssl bug
On Mon, 3 Dec 2001, Herold Heiko wrote: However it still won' work due to mising gen_sslfunc.obj : error LNK2001: unresolved external symbol _lrand48 gen_sslfunc.obj : error LNK2001: unresolved external symbol _srand48 In fact looking through the VC++ documentation there doesn't seem to be a rand48() available. You can replace those onse with just plain rand() and srand() or just write your own. No one in their right mind shall ever depend on using that random value anyway for anything that pretends to be secure, as that wouldn't be considered random enough, and thus it is not secure enough to use for seeding SSL. If /dev/urandom or similar isn't present, a serious SSL-using person should provide a good random source himself. If not, there's not much difference in using lrand48() or just plain rand(). They're both software (faked) random generators. For windows users, I recommend using the RAND_screen() function (it is only presen in windows), it uses the current displayed screen as input to the random engine. Of course, I may be completely wrong. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Wget 1.8-beta2 now available
On 2001-12-01 23:30 +0100, Hrvoje Niksic wrote: Here is the next 1.8 beta. Please test it if you can -- try compiling it on your granma's Ultrix box, run it on your niece's flashy web site, see if cookies work, etc. Get it from: ftp://gnjilux.srk.fer.hr/pub/unix/util/wget/.betas/wget-1.8-beta2.tar.gz Success: - Debian GNU/Linux woody, 80x86, GCC 2.95.4 - Solaris 7, SPARC, GCC 2.95.2 Failure: - HP-UX 10.0, PA-RISC, GCC 3.0.1 Problem #1 : gcc -I. -I.-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\ -O2 -Wall -Wno-implicit -c connect.c connect.c: In function `test_socket_open': connect.c:190: warning: passing arg 2 of `select' from incompatible pointer type connect.c: In function `select_fd': connect.c:283: warning: passing arg 2 of `select' from incompatible pointer type connect.c:283: warning: passing arg 3 of `select' from incompatible pointer type connect.c:283: warning: passing arg 4 of `select' from incompatible pointer type (These are just warnings.) Problem #2 : gcc -I. -I.-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\ -O2 -Wall -Wno-implicit -c host.c host.c: In function `lookup_host': host.c:258: `h_errno' undeclared (first use in this function) host.c:258: (Each undeclared identifier is reported only once host.c:258: for each function it appears in.) Apparently, h_errno is not declared at all under HP-UX (ie. find /usr/include -follow -type f | xargs grep h_errno turns up nothing). Declaring h_errno (extern int h_errno;) fixes the problem. I suppose we need something like : #if HPUX extern int h_errno; #endif Problem #3 : gcc -I. -I.-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\ -O2 -Wall -Wno-implicit -c snprintf.c snprintf.c: In function `dopr': snprintf.c:311: `short int' is promoted to `int' when passed through `...' snprintf.c:311: (so you should pass `int' not `short int' to `va_arg') snprintf.c:323: `short unsigned int' is promoted to `int' when passed through `...' snprintf.c:335: `short unsigned int' is promoted to `int' when passed through `...' snprintf.c:349: `short unsigned int' is promoted to `int' when passed through `...' GCC has become very annoying with that sort of things... I did the suggested changes and the error messages vanished. - OSF/1 4.0, alpha, DEC C 5.6 Problem #1 : cc -std1 -I. -I.-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\ -O -Olimit 2000 -c host.c cc: Error: host.c, line 221: In the initializer for lst[0], tmpstore does not have a constant address, but occurs in a context that requires an address constant. This is an extension of the language. char *lst[] = { tmpstore, NULL }; --^ The error message is misleading, IMO. The real problem is that we're initialising an auto array, which is something C does not support, at least not C89/C90. The following patch silences the compiler : diff -ur wget-1.8-beta2/src/host.c wget-1.8-beta2_aym/src/host.c --- wget-1.8-beta2/src/host.c Fri Nov 30 11:50:29 2001 +++ wget-1.8-beta2_aym/src/host.c Mon Dec 3 16:30:58 2001 @@ -218,7 +218,7 @@ if ((int)addr != -1) { char tmpstore[IP4_ADDRESS_LENGTH]; - char *lst[] = { tmpstore, NULL }; + char *lst[2]; /* ADDR is defined to be in network byte order, which is what this returns, so we can just copy it to STORE_IP. However, @@ -232,6 +232,8 @@ offset = 0; #endif memcpy (tmpstore, (char *)addr + offset, IP4_ADDRESS_LENGTH); + lst[0] = tmpstore; + lst[1] = NULL; return address_list_new (lst); } Problem #2 : There is also this shit. Take a deep breath : cc: Warning: snprintf.c, line 128: In this declaration, type signed long long is a language extension. LLONG value, int base, int min, int max, int flags); ---^ cc: Warning: snprintf.c, line 170: In this declaration, type signed long long is a language extension. LLONG value; --^ cc: Warning: snprintf.c, line 315: In this statement, type signed long long is a language extension. value = va_arg (args, LLONG); --^ cc: Warning: snprintf.c, line 315: In this statement, type signed long long is a language extension. value = va_arg (args, LLONG); --^ cc: Warning: snprintf.c, line 315: In this statement, type signed long long is a language extension. value = va_arg (args, LLONG); --^ cc: Warning: snprintf.c, line 315: In this statement, type signed long long is a language extension.
Re: Wget 1.8-beta2 now available
On Mon, 3 Dec 2001, Andre Majorel wrote: Problem #2 : gcc -I. -I.-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\ -O2 -Wall -Wno-implicit -c host.c host.c: In function `lookup_host': host.c:258: `h_errno' undeclared (first use in this function) host.c:258: (Each undeclared identifier is reported only once host.c:258: for each function it appears in.) Apparently, h_errno is not declared at all under HP-UX (ie. find /usr/include -follow -type f | xargs grep h_errno turns up nothing). Declaring h_errno (extern int h_errno;) fixes the problem. I suppose we need something like : #if HPUX extern int h_errno; #endif Better yet: #if !HAVE_DECL_H_ERRNO extern int h_errno; #endif and use AC_CHECK_DECLS(h_errno,,,[#include netdb.h]) somewhere in configure.in. -- + Maciej W. Rozycki, Technical University of Gdansk, Poland + +--+ +e-mail: [EMAIL PROTECTED], PGP key available+
Re: Make -p work with framed pages.
Ian Abbott [EMAIL PROTECTED] writes: On 1 Dec 2001 at 4:04, Hrvoje Niksic wrote: As a TODO entry summed up: * -p should probably go _two_ more hops on FRAMESET pages. More generally, I think it probably needs to be made to work for nested framesets too. Maybe. You can make it work by removing the depth checking in line 259 of recur.c. I was uneasy with that because of the fact that, by the time this check is evaluated, we don't really know whether the current URL comes from a FRAMESET. We only know that the depth has been exceeded and that we're only accepting requisites. So if a, say, image file, happens to serve text/html, perhaps due to it being an error page, we'll follow its requisites. The exact same thing could happen with buggy framesets. So I opted to allow only one additional recursion level, as the TODO item specified. I suppose we could allow another one, in case a frame contains more frames, but we should probably not make it unlimited. Even with `-np', there shouldn't be a way for bogus external output to make Wget completely ignore the maximum download depth the user has specified.
RE: Win ssl bug
Sorry, I forgot. Changelog: * windows\Makefile.src: add gen_sslfunc.c * windows\Makefile.src.bor: ditto. Heiko -- -- PREVINET S.p.A.[EMAIL PROTECTED] -- Via Ferretto, 1ph x39-041-5907073 -- I-31021 Mogliano V.to (TV) fax x39-041-5907087 -- ITALY
Re: Wget 1.8-beta2 now available
On 2001-12-03 18:30 +0100, Hrvoje Niksic wrote: Andre Majorel [EMAIL PROTECTED] writes: gcc -I. -I.-DHAVE_CONFIG_H -DSYSTEM_WGETRC=\/usr/local/etc/wgetrc\ -DLOCALEDIR=\/usr/local/share/locale\ -O2 -Wall -Wno-implicit -c connect.c connect.c: In function `test_socket_open': connect.c:190: warning: passing arg 2 of `select' from incompatible pointer type connect.c: In function `select_fd': connect.c:283: warning: passing arg 2 of `select' from incompatible pointer type connect.c:283: warning: passing arg 3 of `select' from incompatible pointer type connect.c:283: warning: passing arg 4 of `select' from incompatible pointer type (These are just warnings.) And weird ones, too. These arguments are of type pointer to fd_set. What would HPUX like to see there? HP-UX 10 wants (int *). However it defines fd_set as struct { long[]; } so it works anyway. HP-UX 10 is wrong. SUS2 (and POSIX ?) say (fd_set *). HP-UX 11 has it right. I suppose the best thing to do is to ignore those warnings. I think I'll use something like: #ifndef h_errno extern int h_errno; #endif h_errno is not necessarily a macro ! What do you think of Maciej's proposal ? Two questions here: * Does HPUX really not have snprintf()? It sounds weird that a modern OS wouldn't have it. I find describing HP-UX 10 as a modern OS mildly amusing. :-) I completely disagree with your perception that snprintf() is to be taken for granted. It's only since C99 that's it's part of C. But to answer your question, no HP-UX doesn't have it (neither in the headers nor in libc). * short int is promoted to int, ok. Does that go for all the architectures, or just some? Should I simply replace short int with int to get it to compile? Yes, replace short int and unsigned short by int. It's not architecture specific, the same thing happened to me on x86. GCC 2.95 doesn't care, GCC 2.96 and 3.0 complain. The error message is misleading, IMO. The real problem is that we're initialising an auto array, which is something C does not support, at least not C89/C90. Indeed. I wonder why I thought that was legal C. Ok, I'll apply your patch. Not enough cafeine. :-) -- André Majorel URL:http://www.teaser.fr/~amajorel/ (Not speaking for my employer, etc.)
Re: Wget 1.8-beta2 now available
Andre Majorel [EMAIL PROTECTED] writes: And weird ones, too. These arguments are of type pointer to fd_set. What would HPUX like to see there? HP-UX 10 wants (int *). However it defines fd_set as struct { long[]; } so it works anyway. HP-UX 10 is wrong. SUS2 (and POSIX ?) say (fd_set *). HP-UX 11 has it right. I suppose the best thing to do is to ignore those warnings. Agreed. I think I'll use something like: #ifndef h_errno extern int h_errno; #endif h_errno is not necessarily a macro ! I know that. The above code says: if h_errno is a macro, do nothing. otherwise, declare it. What do you think of Maciej's proposal ? It sounds like the right thing to do. Two questions here: * Does HPUX really not have snprintf()? It sounds weird that a modern OS wouldn't have it. I find describing HP-UX 10 as a modern OS mildly amusing. :-) How old is it? I used to work on HPUX 9, and I'm not old by most definitions of the word. I completely disagree with your perception that snprintf() is to be taken for granted. It's only since C99 that's it's part of C. It's been a part of C since C99, that's true. But Wget relies on a lot of functionality not strictly in C, from alloca to the socket interface. Also, snprintf has become a big security thing recently, when a number of exploits was based on overflowing a buffer written to by sprintf. The pressure on vendors might be responsible for some of them being unusually swift in providing the function. But yes, I know I can't take it for granted, hence the provided replacement. But to answer your question, no HP-UX doesn't have it (neither in the headers nor in libc). HPUX 11 doesn't have it either? Interesting. * short int is promoted to int, ok. Does that go for all the architectures, or just some? Should I simply replace short int with int to get it to compile? Yes, replace short int and unsigned short by int. It's not architecture specific, the same thing happened to me on x86. GCC 2.95 doesn't care, GCC 2.96 and 3.0 complain. Ah, ok. Thanks for the info.
Re: Wget 1.8-beta2 now available
On 2001-12-01 23:30 +0100, Hrvoje Niksic wrote: Here is the next 1.8 beta. Please test it if you can -- try compiling it on your granma's Ultrix box, run it on your niece's flashy web site, see if cookies work, etc. Get it from: ftp://gnjilux.srk.fer.hr/pub/unix/util/wget/.betas/wget-1.8-beta2.tar.gz Success: - NCR MP-RAS 3.0, x86, NCR High Performance C Compiler R3.0c - FreeBSD 4.0, x86, GCC 2.95.2 Thanks ! -- André Majorel URL:http://www.teaser.fr/~amajorel/ (Not speaking for my employer, etc.)
Re: Wget 1.8-beta2 now available
Maciej W. Rozycki [EMAIL PROTECTED] writes: Better yet: #if !HAVE_DECL_H_ERRNO extern int h_errno; #endif and use AC_CHECK_DECLS(h_errno,,,[#include netdb.h]) somewhere in configure.in. My version of Autoconf does not have an AC_CHECK_DECLS macro.
Re: Wget 1.8-beta2 now available
On Mon, 3 Dec 2001, Hrvoje Niksic wrote: and use AC_CHECK_DECLS(h_errno,,,[#include netdb.h]) somewhere in configure.in. My version of Autoconf does not have an AC_CHECK_DECLS macro. Hmm, how about considering autoconf 2.52? It is said to be less broken than 2.13 and indeed it seems so. If it's too late for such a transition, which I suppose it is (admittedly, the jump from 1.7.1 to 1.8 was done an express way), you may just borrow the macro from autoconf until the transition is done. This way it's done the compatible way. I may see how to do this, but I have no free time slots at the moment, so I may be unable to code anything until the next week. Anyone feel free to do it sooner. Maciej -- + Maciej W. Rozycki, Technical University of Gdansk, Poland + +--+ +e-mail: [EMAIL PROTECTED], PGP key available+
Re: Wget 1.8-beta2 now available
On 2001-12-03 19:16 +0100, Hrvoje Niksic wrote: I find describing HP-UX 10 as a modern OS mildly amusing. :-) How old is it? I used to work on HPUX 9, and I'm not old by most definitions of the word. Around 1995. I completely disagree with your perception that snprintf() is to be taken for granted. It's only since C99 that's it's part of C. It's been a part of C since C99, that's true. But Wget relies on a lot of functionality not strictly in C, from alloca to the socket interface. Also, snprintf has become a big security thing recently, when a number of exploits was based on overflowing a buffer written to by sprintf. The pressure on vendors might be responsible for some of them being unusually swift in providing the function. But yes, I know I can't take it for granted, hence the provided replacement. Yes, I'm with you on that. We have exactly the same problems as you here and I for one wish snprintf() had been there from the start. But to answer your question, no HP-UX doesn't have it (neither in the headers nor in libc). HPUX 11 doesn't have it either? Interesting. HP-UX 10 doesn't but HP-UX 11 has it, according to docs.hp.com. The work you did on the list of already downloaded URLs seems to have been efficient ; Wget's long standing tendency to forget files in recursive downloads appears to be gone. A million thanks to Hrvoje, the contributors and the testers. -- André Majorel URL:http://www.teaser.fr/~amajorel/ (Not speaking for my employer, etc.)
Re: Wget 1.8-beta2 now available
Maciej W. Rozycki [EMAIL PROTECTED] writes: On Mon, 3 Dec 2001, Hrvoje Niksic wrote: and use AC_CHECK_DECLS(h_errno,,,[#include netdb.h]) somewhere in configure.in. My version of Autoconf does not have an AC_CHECK_DECLS macro. Hmm, how about considering autoconf 2.52? Yes, but not before the 1.8 release. I'd like this week to be a stabilizing period, with no new features or instabilities. It is said to be less broken than 2.13 and indeed it seems so. Autoconf 2.13 works for Wget, so I'm not all that eager to replace it. Let them stabilize 2.52 some more. I have no desire to be mutilated by the bleeding edge of Autoconf development. you may just borrow the macro from autoconf until the transition is done. This way it's done the compatible way. I may see how to do this, but I have no free time slots at the moment, so I may be unable to code anything until the next week. Anyone feel free to do it sooner. IMO there is no reason for that. The solution from Wget 1.7.1 may not be pretty, but it worked on all known architectures without a single bug report about it: #ifndef h_errno # ifndef __CYGWIN__ extern int h_errno; # endif #endif That should be good enough for 1.8.
Wget 1.8-beta3 now available
Bugfixes since 1.8-beta2. Please test it from clean compilation on Unix (Windows and MacOS are known not to compile without modifications when SSL is used.) Get it from: ftp://gnjilux.srk.fer.hr/pub/unix/util/wget/.betas/wget-1.8-beta3.tar.gz (The `.betas' directory is intentionally unreadable, but the file is there.)
Re: Wget 1.8-beta3 now available
On 2001-12-03 21:55 +0100, Hrvoje Niksic wrote: Bugfixes since 1.8-beta2. Please test it from clean compilation on Unix (Windows and MacOS are known not to compile without modifications when SSL is used.) Get it from: ftp://gnjilux.srk.fer.hr/pub/unix/util/wget/.betas/wget-1.8-beta3.tar.gz This one compiles on all platforms. Solaris 7 OK FreeBSD 4.0 OK HP-UX 10OK MP-RAS 3.0 OK Debian Linux woody OK OSF/1 4.0 OK Beautiful. :-) -- André Majorel URL:http://www.teaser.fr/~amajorel/ (Not speaking for my employer, etc.)
Re: Wget 1.8-beta3 now available
You can add (not using ssl) Irix64 6.5.12m SunOS 5.8 ... but at home I get gen-md5.c:31: md5.h: No such file or directory when I try to compile the newer cvs versions. (debian/potato, openssl installed in /usr/local) On Mon, 3 Dec 2001 22:47:32 +0100 Andre Majorel [EMAIL PROTECTED] wrote: On 2001-12-03 21:55 +0100, Hrvoje Niksic wrote: Bugfixes since 1.8-beta2. Please test it from clean compilation on Unix (Windows and MacOS are known not to compile without modifications when SSL is used.) Get it from: ftp://gnjilux.srk.fer.hr/pub/unix/util/wget/.betas/wget-1.8-beta3.tar.gz This one compiles on all platforms. Solaris 7 OK FreeBSD 4.0 OK HP-UX 10 OK MP-RAS 3.0OK Debian Linux woodyOK OSF/1 4.0 OK Beautiful. :-) -- André Majorel URL:http://www.teaser.fr/~amajorel/ (Not speaking for my employer, etc.)
patch for hanging on connect
I have a problem where my wget hangs on the connect call. found with an strace.Someone submitted a patch in 1999 for this problem so I was wondering if we can get the patch in the source? here is the patch I made for the current release Not clean some unsed variables; but I was cutting and pasting. /src/connect.c 21c21 #include unistd.h --- 46,47c46 #include signal.h #include sys/types.h --- 55,62c54 int connect_alarmed; int err = 0; struct sigaction timeout_on_connect; sigset_t masked_signals; static void connect_alarm_occurred (int data) { connect_alarmed = 1; } --- 107,125d98 sigfillset (masked_signals); timeout_on_connect.sa_handler = connect_alarm_occurred; timeout_on_connect.sa_mask = masked_signals; timeout_on_connect.sa_flags = SA_ONESHOT; timeout_on_connect.sa_restorer = NULL; connect_alarmed = 0; if (sigaction (SIGALRM, timeout_on_connect, NULL) == -1) { DEBUGP ((Couldn`t setup a SIGALRM handler, have to use the default)); } else { alarm (opt.timeout); } 133d105 alarm(0); Thanks Rick Herbel PS This would have saved me some debug time :(