While Google tends to behave better in regards to user-agent shenanigans, I’ve even had to drop-route them before when they started raping some of my sites. It seems to be better these days, but search spiders tend to be oblivious to how much traffic they can drive, especially when they malfunction and decide to download a large file repeatedly for days on end.
-dosman > On Sep 6, 2017, at 10:14 AM, Joshua Knapp <[email protected]> wrote: > > They mean that some times it didn't identify as yandex, sometimes it would > fake as a browser. I saw the same thing in the logs after I started trying > to block the bot by user agent. > > On Sep 6, 2017 7:09 AM, "Claes Wallin (韋嘉誠)" > <[email protected] > <mailto:[email protected]>> wrote: > On Sep 6, 2017 4:09 PM, "Ken Fallon" <[email protected] <mailto:[email protected]>> > wrote: > http://www.webhostingtalk.com/showthread.php?t=924727 > <http://www.webhostingtalk.com/showthread.php?t=924727> > > That thread is so funny. People really don't bother to read the full tread > before they post, do they? > > - Using robots.txt didn't help, so here's how I detect and redirect that bot. > - Dude use robots.txt > - IP block! > - I redirected instead. > - Dude you really should be using robots txt. > - Here's how I IP block! > > :-D > > Some people seem to believe that text file has magical powers. > > > I didn't get this part: > > > One more thing: it identifies as client, not as bot !! Google and Yahoo > > identified as bots ! > > What are they talking about? Is there anything aside from the user agent that > indicates what the client is? > > -- > /c > > _______________________________________________ > Hpr mailing list > [email protected] <mailto:[email protected]> > http://hackerpublicradio.org/mailman/listinfo/hpr_hackerpublicradio.org > <http://hackerpublicradio.org/mailman/listinfo/hpr_hackerpublicradio.org> > > _______________________________________________ > Hpr mailing list > [email protected] > http://hackerpublicradio.org/mailman/listinfo/hpr_hackerpublicradio.org
_______________________________________________ Hpr mailing list [email protected] http://hackerpublicradio.org/mailman/listinfo/hpr_hackerpublicradio.org
