Follow-up Comment #5, bug #30999 (project wget):
I have read the robots.txt spec thoroughly and found no way to set crawl-delay
for a specific file. If someone could look into it that would be nice.
Otherwise I think the best solution is to set --wait to the matching
crawl-delay if the user hasn't set --wait already.
_______________________________________________________
Reply to this item at:
<http://savannah.gnu.org/bugs/?30999>
_______________________________________________
Message sent via/by Savannah
http://savannah.gnu.org/
