On Mon, 22 Sep 2003, Tyler Mitchell wrote: > I find the official web site that isn't recursive sucking friendly and > wondered if anyone has any ideas that might help. > Here is one that doesn't have .html extensions - is that the problem? > http://www.bbc.co.uk/dna/h2g2/ > Any ideas?
some sample output would have helped. regardless, the problem is that the site owners don't want you to do what you want to do. in: $ wget --debug --mirror http://www.bbc.co.uk/dna/h2g2/ |& tail you'll notice: no-follow in www.bbc.co.uk/dna/h2g2/index.html: 1 $ grep -ie nofollow www.bbc.co.uk/dna/h2g2/index.html www.bbc.co.uk/dna/h2g2/index.html:<meta name="robots" content="nofollow"> perhaps the no-follow error should not just be sent in debug. /a
