-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Srinivasan Palaniappan wrote: > I am using WGET version 1.10.2, and trying to crawl through a secured > site (that we are developing for our customer) I noticed two things. > WGET is not downloading all the binaries in the website. It downloads > about 30% of it then skips the rest of the documents. But I don't see > any log files that shows me some kind of error messaging saying unable > to download during spidering, I am not sure I am doing the right thing > can you let me know from the following .wgetrc file and the command line > I run. > > > > .wgetrc > > > exclude_directories = > /ascp/commerce/catalog,/ascp/commerce/checkout,/ascp/commerce/user,/ascp/commerce/common,/ascp/commerce/javascript,/ascp/commerce/css > > > include_directories = /ascp/commerce,/ascp/commerce/scp/downloads > > dir_prefix=\spiderfiles\ascpProd\wget > > domains=www.mysite.com > > no_parent=on > > secure-protocol=SSLv3
^^^ This should use an underscore, not a dash. > wget -r l5 --save-headers --no-check-certificate https://www.mystie.com ^^ - -r doesn't take an argument. Perhaps you wanted a -l before the 15? > In addition, I noticed when the metadata information written to the > downloaded file has only HTTP has scheme, which is somewhat weird do you > know anything about it? I'm not understanding you here. Do you mean that it said, "https://...: Unsupported scheme"? In that case, I don't see how it could have downloaded 30% of anything, as it means it wasn't compiled with support for SSL and HTTPS. The best way to try to see what might be going on, is to invoke wget with the --debug flag, and probably use the "-o logfile" option. That could help us to see what might be going on. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHYCXx7M8hyUobTrERAs4AAJsFXHLNnV/9hmtNNd03tR8jlCswkwCeP7eA wKWaIMY2XZk5vwP4RK0eVo8= =rPh2 -----END PGP SIGNATURE-----
