perl auto-downloader thingy and wget
Hi, Im writing a perl auto-downloader that is supposed to maintain the directory structure. My problem is that with a link like http://www.cisco.com/abcd extracted from a file i have no know no way of knowing whether what i am downloading is a file or directory(because when you try to d/l a dir the server sends the default page). As a result i dont know whther to store m the file the server sent with abcd or as xyz.html. Is it true that when you request a dir www.cisco.com/abcd the server sends a redirect to www.cisco.com/abcd/ .How are you handling this problem. Please advice. Bye
Re: perl auto-downloader thingy and wget
Jenny Eastman <[EMAIL PROTECTED]> writes: > Im writing a perl auto-downloader that is supposed to maintain the > directory structure. My problem is that with a link like > http://www.cisco.com/abcd extracted from a file i have no know no > way of knowing whether what i am downloading is a file or > directory(because when you try to d/l a dir the server sends the > default page). As a result i dont know whther to store m the file > the server sent with abcd or as xyz.html. > > Is it true that when you request a dir www.cisco.com/abcd the server > sends a redirect to www.cisco.com/abcd/ .How are you handling this > problem. Wget handles this by changing the file name to abcd/index.html if/when the server sends the redirect. Therefore the file "abcd" will be saved as "abcd", while a directory "abcd" will be saved as "abcd/index.html".
Greetings, and a question
Greetings! I'm new to this mailing list, though not to wget. I've successfully compiled wget on a variety of platforms in the past, but I seem to be having a problem, and I'm hoping someone can help me. I am attempting to build wget 1.8.2 on one of the new G5 PowerMac machines (dual processor) from Apple. I also compiled OpenSSL 0.9.7c and installed it; the tests all ran without a hitch, so I'm assuming that OpenSSL is compiled and running correctly. When building wget, the link phase bombs with the following error message: gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o cookies.o fnmatch.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o snprintf.o gen_sslfunc.o url.o utils.o version.o -L/usr/local/lib -lssl -lcrypto ld: Undefined symbols: _OPENSSL_add_all_algorithms_noconf make[1]: *** [wget] Error 1 make: *** [src] Error 2 I've successfully built wget with OpenSSL support previously on a G4 PowerMac, so I'm wondering if there's an issue with the G5's compiler (gcc 3.3 with G5 optimizations) or if there's something else wrong. Perhaps I forgot to configure something correctly with OpenSSL or wget? Any suggestions? Best Regards, Rob Poole [EMAIL PROTECTED]
Compile and link problems with wget 1.9 beta5
After ploughing through the archives of this mailing list, looking for additional clues why wget 1.8.2 wasn't linking correctly, I found that wget 1.9 beta 5 was released recently. I downloaded the source code for wget 1.9 beta 5 and am getting the same link problems I was getting with 1.8.2: /bin/sh ../libtool --mode=link gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o convert.o cookies.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o snprintf.o gen_sslfunc.o url.o utils.o version.o -lssl -lcrypto mkdir .libs gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o convert.o cookies.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o snprintf.o gen_sslfunc.o url.o utils.o version.o -lssl -lcrypto ld: Undefined symbols: _OPENSSL_add_all_algorithms_noconf make[1]: *** [wget] Error 1 make: *** [src] Error 2 I've tried to determine if my OpenSSL installation was built wrong, but as far as I can determine, it's OK. That doesn't mean that there's nothing wrong with OpenSSL on this platform, but so far, this link error has been the only problem I've encountered. The platform is a dual-processor G5 running Mac OS X 10.2.8 with the latest developer tools installed (gcc 3.3 with G5 optimizer settings available, although I haven't used any of the command line switches to turn those optimizations on). Help? Best Regards, Rob Poole [EMAIL PROTECTED]
RE: Compile and link problems with wget 1.9 beta5
Do you see the missing symbol when you do an "nm -D" command against either libssl.so or libcrypto.so? (It shows up on my Linux system in libcrypto.so.) Mark Post -Original Message- From: Robert Poole [mailto:[EMAIL PROTECTED] Sent: Sunday, October 12, 2003 2:23 PM To: [EMAIL PROTECTED] Subject: Compile and link problems with wget 1.9 beta5 After ploughing through the archives of this mailing list, looking for additional clues why wget 1.8.2 wasn't linking correctly, I found that wget 1.9 beta 5 was released recently. I downloaded the source code for wget 1.9 beta 5 and am getting the same link problems I was getting with 1.8.2: /bin/sh ../libtool --mode=link gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o convert.o cookies.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o snprintf.o gen_sslfunc.o url.o utils.o version.o -lssl -lcrypto mkdir .libs gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o convert.o cookies.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o snprintf.o gen_sslfunc.o url.o utils.o version.o -lssl -lcrypto ld: Undefined symbols: _OPENSSL_add_all_algorithms_noconf make[1]: *** [wget] Error 1 make: *** [src] Error 2 I've tried to determine if my OpenSSL installation was built wrong, but as far as I can determine, it's OK. That doesn't mean that there's nothing wrong with OpenSSL on this platform, but so far, this link error has been the only problem I've encountered. The platform is a dual-processor G5 running Mac OS X 10.2.8 with the latest developer tools installed (gcc 3.3 with G5 optimizer settings available, although I haven't used any of the command line switches to turn those optimizations on). Help? Best Regards, Rob Poole [EMAIL PROTECTED]
Re: Compile and link problems with wget 1.9 beta5
"Robert Poole" <[EMAIL PROTECTED]> writes: > gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o convert.o > cookies.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o > headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o > gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o > snprintf.o gen_sslfunc.o url.o utils.o version.o -lssl -lcrypto > ld: Undefined symbols: > _OPENSSL_add_all_algorithms_noconf As far as I can tell, the call to SSLeay_add_all_algorithms in gen_sslfunc.c expands to OPENSSL_add_all_algorithms_noconf (without the leading underscore). I really don't know why it fails to link for you; if I had to guess, I'd try checking two things. It could be a mismatch between OpenSSL headers and the installed libraries. Please check that the headers Wget is picking up and the libraries it's linking with are from the same OpenSSL version. Or, it could be a problem with libtool, which Wget uses to be able to (easily) link with external libraries. Try running the link line manually, outside make and without libtool. See if linking works then. For better or worse, noone has reported this problem so far. It's either specific to your architecture or to your setup. Do other SSL programs, such as curl, compile successfully?
Re: Compile and link problems with wget 1.9 beta5
First off, on OS X (which is a BSD-based system, so a lot of the conventions don't match up with Linux), OpenSSL builds libcrypto.a and libssl.a and stuffs them in /usr/local/lib (and also creates a /usr/local/ssl/lib, which is empty -- go figure). Secondly, nm doesn't take a -D command line option for us. However, I did a "nm libcrypto.a" and found the following (after grepping for OPENSSL to weed out what I didn't care about): T _OPENSSL_add_all_algorithms_conf U _OPENSSL_add_all_algorithms_noconf So the symbol is there in libcrypto but has an undefined value, if I'm reading this right. So I guess this begs the question, what next? Best Regards, Rob Poole On Sunday, October 12, 2003, at 12:52 PM, Post, Mark K wrote: Do you see the missing symbol when you do an "nm -D" command against either libssl.so or libcrypto.so? (It shows up on my Linux system in libcrypto.so.) Mark Post -Original Message- From: Robert Poole [mailto:[EMAIL PROTECTED] Sent: Sunday, October 12, 2003 2:23 PM To: [EMAIL PROTECTED] Subject: Compile and link problems with wget 1.9 beta5 After ploughing through the archives of this mailing list, looking for additional clues why wget 1.8.2 wasn't linking correctly, I found that wget 1.9 beta 5 was released recently. I downloaded the source code for wget 1.9 beta 5 and am getting the same link problems I was getting with 1.8.2: /bin/sh ../libtool --mode=link gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o convert.o cookies.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o snprintf.o gen_sslfunc.o url.o utils.o version.o -lssl -lcrypto mkdir .libs gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o convert.o cookies.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o snprintf.o gen_sslfunc.o url.o utils.o version.o -lssl -lcrypto ld: Undefined symbols: _OPENSSL_add_all_algorithms_noconf make[1]: *** [wget] Error 1 make: *** [src] Error 2 I've tried to determine if my OpenSSL installation was built wrong, but as far as I can determine, it's OK. That doesn't mean that there's nothing wrong with OpenSSL on this platform, but so far, this link error has been the only problem I've encountered. The platform is a dual-processor G5 running Mac OS X 10.2.8 with the latest developer tools installed (gcc 3.3 with G5 optimizer settings available, although I haven't used any of the command line switches to turn those optimizations on). Help? Best Regards, Rob Poole [EMAIL PROTECTED]
Re: Compile and link problems with wget 1.9 beta5
I did a little digging. Even though I compiled openssl, and it installed things under /usr/local, there are other libssl and libcrypto versions on the system; so I'm kind of scratching my head, trying to figure out how to force wget to build against the stuff that I actually compiled, instead of something else. I can only guess that somehow the linker is finding the wrong libraries and is blowing up when it tries to link against them. I examined libssl.a and libcrypto.a, and the _ OPENSSL_add_all_algorithms_noconf symbol is defined in libcrypto.a that was installed when I built openssl. However, there's a libssl.so and a libcrypto.so elsewhere on the system, and a libssl.dylib and libcrypto.dylib elsewhere as well... It's been quite some time since I've had to dig through a linker problem like this, so I'm kind of lost. I'd be willing to let someone I could trust ssh into my system and dig around and see what they see. As far as I can tell, the only headers for openssl are the ones that I installed with openssl 0.9.7c. I'm trying to search on the net, but nobody seems to be compiling the code from scratch; almost everyone is using fink. Best Regards, Rob Poole [EMAIL PROTECTED] On Sunday, October 12, 2003, at 01:51 PM, Hrvoje Niksic wrote: "Robert Poole" <[EMAIL PROTECTED]> writes: gcc -O2 -Wall -Wno-implicit -o wget cmpt.o connect.o convert.o cookies.o ftp.o ftp-basic.o ftp-ls.o ftp-opie.o getopt.o hash.o headers.o host.o html-parse.o html-url.o http.o init.o log.o main.o gen-md5.o netrc.o progress.o rbuf.o recur.o res.o retr.o safe-ctype.o snprintf.o gen_sslfunc.o url.o utils.o version.o -lssl -lcrypto ld: Undefined symbols: _OPENSSL_add_all_algorithms_noconf As far as I can tell, the call to SSLeay_add_all_algorithms in gen_sslfunc.c expands to OPENSSL_add_all_algorithms_noconf (without the leading underscore). I really don't know why it fails to link for you; if I had to guess, I'd try checking two things. It could be a mismatch between OpenSSL headers and the installed libraries. Please check that the headers Wget is picking up and the libraries it's linking with are from the same OpenSSL version. Or, it could be a problem with libtool, which Wget uses to be able to (easily) link with external libraries. Try running the link line manually, outside make and without libtool. See if linking works then. For better or worse, noone has reported this problem so far. It's either specific to your architecture or to your setup. Do other SSL programs, such as curl, compile successfully?
Re: some wget patches against beta3
On Tuesday 07 of October 2003 12:08, Hrvoje Niksic wrote: > Thanks! btw. looking into test4 I see that autoconf conception is used in weird way. Normally aclocal.m4 is autogenerated by aclocal command (and it includes sources of macros from /usr/share/autoconf/autoconf/*.m4 and from local acinclude.m4). You are seem to put all macros (even local macros!) inside of aclocal.m4. This is bad because it doesn't allow to regenerate autoconf/aclocal stuff in case when it's needed. aclocal command will overwrite aclocal.m4 and all local macros will be lost. We at PLD are right now patching wget to move all local macros into acinclude.m4 but it would be better if that was done properly at wget maintainers level :) ps. I see that beta5 is there so time to look into this one... argh, same things as in test4 -- Arkadiusz MiĆkiewiczCS at FoE, Wroclaw University of Technology arekm.pld-linux.org AM2-6BONE, 1024/3DB19BBD, arekm(at)ircnet, PLD/Linux
Re: Compile and link problems with wget 1.9 beta5
"Robert Poole" <[EMAIL PROTECTED]> writes: > I did a little digging. Even though I compiled openssl, and it > installed things under /usr/local, there are other libssl and > libcrypto versions on the system; so I'm kind of scratching my head, > trying to figure out how to force wget to build against the stuff > that I actually compiled, instead of something else. If I remember correctly, OpenSSL installs under /usr/local/ssl by default. Wget, on the other hand, tries to be a decent citizen and first attempt to look for libraries in the standard locations, where "standard" means whatever is in the compiler's include/library search path. To quote configure.in: dnl OpenSSL's root was not specified, so we have to guess. First dnl try the system default location, then "/usr/local/ssl" (where dnl OpenSSL installs by default), then "/usr/local" (traditional dnl choice for installation root), then "/opt". ssl_all_roots="system-default /usr/local/ssl /usr/local /opt" You can specify OpenSSL root with `--with-ssl=ROOT'. For example, `./configure --with-ssl=/usr/local/ssl' will prefer the libraries in /usr/local/ssl/lib and includes in /usr/local/ssl/include over any other. > I examined libssl.a and libcrypto.a, and the _ > OPENSSL_add_all_algorithms_noconf symbol is defined in libcrypto.a > that was installed when I built openssl. However, there's a > libssl.so and a libcrypto.so elsewhere on the system, and a > libssl.dylib and libcrypto.dylib elsewhere as well... [...] > As far as I can tell, the only headers for openssl are the ones that > I installed with openssl 0.9.7c. That almost confirms my suspicion. I'm guessing that Wget links with an older libssl.so (because they're in the compiler's library search path), but finds the includes from the newer version (because those are the only SSL includes on the system, and they're in the compiler's include path). Please try using `./configure --with-ssl=/usr/local' or `./configure --with-ssl=/usr/local/ssl' and see if the problem goes away.
Re: some wget patches against beta3
Arkadiusz Miskiewicz <[EMAIL PROTECTED]> writes: > On Tuesday 07 of October 2003 12:08, Hrvoje Niksic wrote: >> Thanks! > > btw. looking into test4 I see that autoconf conception is used in > weird way. Normally aclocal.m4 is autogenerated by aclocal command "Normally" only if you're also using Automake. Wget doesn't use Automake and therefore has no use for `aclocal'. > You are seem to put all macros (even local macros!) inside of > aclocal.m4. Yes, local macros are in aclocal.m4 -- that's how it got its name. Autoconf manual says: To create a `configure' script with Autoconf, you need to write an Autoconf input file `configure.ac' (or `configure.in') and run `autoconf' on it. If you write your own feature tests to supplement those that come with Autoconf, you might also write files called `aclocal.m4' and `acsite.m4'. Then, later: Those files [`aclocal.m4' and `acsite.m4'] can contain your site's or the package's own Autoconf macro definitions. `acinclude.m4' is not even mentioned, so I must assume it's another Automake thing. > This is bad because it doesn't allow to regenerate autoconf/aclocal The macros in aclocal.m4 are not autogenerated, therefore there is no need for regeneration. Am I missing something? > We at PLD are right now patching wget to move all local macros into > acinclude.m4 but it would be better if that was done properly at > wget maintainers level :) Could you explain the motivation behind the move from aclocal.m4 to acinclude.m4?
Re: How do you pronounce Hrvoje?
Hrvoje and I have had an off-list dialogue about this subject. We've settled on "HUR-voy-eh" as the closest phonetic rendition of his name for English speakers. It helps to remember that the "r" is rolled. Tony
Re: Compile and link problems with wget 1.9 beta5
I tried what you suggested, and of course did a "make clean" to make sure everything got recompiled from scratch, but the link still bombs with the same error message. I even copied the libssl.a and libcrypto.a files to /usr/local/ssl/lib (which was empty up to this point, very strange), and then ran the configure script as follows: ./configure --with-ssl=/usr/local/ssl Still nothing. Hrm. On a hunch, rebuilt OpenSSL without specifying any directories, reconfiguring to use all the default values. This time, /usr/local/ssl/lib has everything you'd expect to be in it. But, wget still won't link correctly. The really frustrating thing is, I had successfully built wget 1.8.2 against openssl 0.9.7 on an iBook and a G4 Cube with a virtually identical setup. The only difference is that the G5 is using an updated version of the dev tools with support for G5-specific optimizations (which I'm not using at the moment). I'm tempted to regress to the older version of the development tools and see if that helps; if so, then the linker got broken in the new version of the development tools. Unfortunately, although I attempted to regress to the previous version of the dev tools (which would roll me back to gcc 3.1), the linking still seems to be broken; so for now, I've returned to using the latest dev tools (gcc 3.3, etc.) and am going to submit a bug report to Apple developer relations. I haven't given up hope of getting this thing built -- but considering that I was able to build and link these packages successfully on a lowly iBook and a G4 Cube, both of which were running the older dev tools suite, I suspect the problem is the newer compiler and linker. Best Regards, Rob Poole On Sunday, October 12, 2003, at 03:55 PM, Hrvoje Niksic wrote: "Robert Poole" <[EMAIL PROTECTED]> writes: I did a little digging. Even though I compiled openssl, and it installed things under /usr/local, there are other libssl and libcrypto versions on the system; so I'm kind of scratching my head, trying to figure out how to force wget to build against the stuff that I actually compiled, instead of something else. If I remember correctly, OpenSSL installs under /usr/local/ssl by default. Wget, on the other hand, tries to be a decent citizen and first attempt to look for libraries in the standard locations, where "standard" means whatever is in the compiler's include/library search path. To quote configure.in: dnl OpenSSL's root was not specified, so we have to guess. First dnl try the system default location, then "/usr/local/ssl" (where dnl OpenSSL installs by default), then "/usr/local" (traditional dnl choice for installation root), then "/opt". ssl_all_roots="system-default /usr/local/ssl /usr/local /opt" You can specify OpenSSL root with `--with-ssl=ROOT'. For example, `./configure --with-ssl=/usr/local/ssl' will prefer the libraries in /usr/local/ssl/lib and includes in /usr/local/ssl/include over any other. I examined libssl.a and libcrypto.a, and the _ OPENSSL_add_all_algorithms_noconf symbol is defined in libcrypto.a that was installed when I built openssl. However, there's a libssl.so and a libcrypto.so elsewhere on the system, and a libssl.dylib and libcrypto.dylib elsewhere as well... [...] As far as I can tell, the only headers for openssl are the ones that I installed with openssl 0.9.7c. That almost confirms my suspicion. I'm guessing that Wget links with an older libssl.so (because they're in the compiler's library search path), but finds the includes from the newer version (because those are the only SSL includes on the system, and they're in the compiler's include path). Please try using `./configure --with-ssl=/usr/local' or `./configure --with-ssl=/usr/local/ssl' and see if the problem goes away.