Hello, On Sun, Nov 03, 2019 at 10:04:46AM -0500, Gene Heskett wrote: > I am developing a list of broken webcrawlers who are repeatedly > downloading my entire web site including the hidden stuff.
[…] > How do I get their attention to stop the DDOS? Or is this a war you > cannot win? Hosting a public web site on a domestic broadband connection with low data transfer allowance isn't the best way to go in 2019, but you can have some success with an escalation of: 1. robots.txt 2. UserAgent banning 3. Fail2Ban and/or apache modules for per-IP quota or requests and bytes. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting