Hi Paul,

sorry for my late answer.

> this will build the folder structure and should get the servers robots.txt 
> file
> edit the file to exclude the other unwanted urls
> then do your actual recursive mirror
> just google example robots.txt to hack what you need

I never thought about this idea. I'll search for it. Thanks for the hint!

Greetings,

Tobias

Reply via email to