A robots.txt file can help with many spiders, along with a link to the
dspace sitemap.
Sitemap: /jspui/sitemap
The robots.txt file can include
Crawl-delay: 10
and it is useful to disallow the search and browse links - eg
Disallow: /jspui/simple-search
May robots get lost in circling
On Thu, Jan 19, 2023 at 11:50:03AM +0100, Florian Wille wrote:
> my DSpace (6.3) Site usually gets around 10k/h requests. This is handled
> quite well. But sometimes there are multiple
> bots/crawlers/spiders/indexers/harvester/whatevers throwing each up to
> 15k/h request at me at the same
Hey There,
my DSpace (6.3) Site usually gets around 10k/h requests. This is handled
quite well. But sometimes there are multiple
bots/crawlers/spiders/indexers/harvester/whatevers throwing each up to
15k/h request at me at the same time and that on top of my 10k/h
standart traffic. This my
