- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: How can I avoid bombarding the servers?
If webmasters have time to put hands to bann this engine, instead of, they could spend it to write Crawl-delay directive, that this engine honors and this described at link provided by default in User-Agent ( http://www.dataparksearch.org/bot ). At other side, DataparkSearch have CrawlDelay command to specify minimal delay between consecutive fetches from same server. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;topic_id=1157474787
