Attempting to mirror a particular web site, with wget 1.8.1, I got many nested directories like .../images/images/images/images etc For example the log file ended like this:
--08:16:37-- http://www.can-online.org.uk/SE/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/spacer.gif => `www.can-online.org.uk/SE/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/images/spacer.gif' Reusing connection to www.can-online.org.uk:80. HTTP request sent, awaiting response... 414 URL Too Long 08:16:37 ERROR 414: URL Too Long. FINISHED --08:16:37-- Downloaded: 23,505,670 bytes in 906 files Similar problems have happened with enough websites that I have tried to mirror, to be worth investigating, so this time I have spent a little time doing this. I'm posting this in case people are interested in the reasons in this particular case. I dont think the exact reasons why these multiple nested directories occur is going to be identical on every website though. Firstly I should say that it is basically caused by the website rather than the excellent wget, but maybe there are ways that wget could be made even better. Also its quite clear that although I think the proposed fix would work in this case, it won't solve all nested directory problems The fundamental problem in this case is that if you ask for a page that does not exist from www.can-online.org.uk, the server does not respond correctly. Instead of presenting the 404 not found, it serves up some default web page. This explains why when wget asks for ridiculous things like http://www.can-online.org.uk/SE/images/images/images/images/images/spacer.gif it receives what appears to be a valid response. However it does not explain why wget asks for things like this in the first place. The reason for that turns out to be that there is a mistake in some of the html pages on the webserver.For example in http://www.can-online.org.uk/contact/ one of the included images has been written as <img src="images/spacer.gif" alt="" width="2"height="1" /> instead of <img src="/images/spacer.gif" alt="" width="2"height="1" /> As a result, wget tried to GET the (non-existent) image http://www.can-online.org.uk/contact/images/spacer.gif Now, because of the fundamental problem mentioned above, instead of returning 404 not found, this web server instead serves up its default html page, instead of an image. Now at this point is the opportunity for wget to be more robust. Because at present, it seems as though when wget sees the html page, it forgets that it was expecting an image and it parses the html and tries to follow all the links in the html. (Note that what is expected must be an image because it is in an <img src=""> construct.) Now, in the case of www.can-online.org.uk the unfortunate fact is that the default html page also contains the same mistake in some of the <img > constructs, so the process repeats ad infinitum with wget trying to get deeper and deeper levels of .../index/index/index/index So my suggestion is this. If wget is following an <img src=""> address from a page, and instead of the expected [image/gif] (or jpeg or whatever) file type the server gives a [text/html], then wget should not follow any links that the text/html file contains. Perhaps you could even argue that it should report an error and not even save the html file, because as far as I can see, it doesn't make any kind of sense to include a [text/html] file where an image should be. Of course this will not necessarily solve the nested directory problem with other websites - because if the mistake in the html was in a link to an html page rather than an image, for example, this fix wouldn't help - but at least it would work on this one. If you are interested in this, I have set up a very simple test case (using 4 files, of a few lines each) on a webserver to demonstrate what wget does, will be happy to post details. Stephen Hewitt
