Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

Randal> Yes, it's possible to configure it so that it works correctly, Randal> but if I recall, I also saw it fetch /cgi/whatever, even though Randal> that was in /robots.txt. I *must* block anything that doesn't Randal> respect /robots.txt. Once they fix that, I might let it loose. Telepor

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> "Michael" == Michael Plump <[EMAIL PROTECTED]> writes: Michael> Teleport Pro, by default, is setup to be a nice little web Michael> robot. Just because one user configures the program to be Michael> evil doesn't mean you should stop other people who are trying Michael> to play nice. And s

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

On 14 Dec 1999, Randal L. Schwartz wrote: > Sounds to me like they are precisely at odds with anyone doing the > kind of blocking that I want to do. That seems like a weird policy, though. nmap, for example, helps people do dastardly things, but that doesn't mean nmap is a bad program; it's how

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> "Eric" == Eric L Brine <[EMAIL PROTECTED]> writes: Eric> Because of #2, Teleport Pro only has one active thread at a time, and it Eric> is idle at least 50% of the time (when downloading image archives). In Eric> other words, it's possible for a user can configure Teleport Pro to Eric> hamm

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> It's also been very successful at catching a whole slew of user-agents > that believe in sucking senselessly. Here's my current block-list: > > [...] > or m{Teleport Pro} # bad robot! > [...] Teleport Pro does have options to control how it behaves: 1. "Obey the Robot Exclusion

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> "Doug" == Doug MacEachern <[EMAIL PROTECTED]> writes: >> > mod_cgi does no waiting. :) The only wait appears to be in the cleanup >> >> yes it does, but after mod_perl cleanups are run. see free_proc_chain() >> in alloc.c Doug> oh wait, I just said what you said, nevermind, I skimmed t

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> > mod_cgi does no waiting. :) The only wait appears to be in the cleanup > > yes it does, but after mod_perl cleanups are run. see free_proc_chain() > in alloc.c oh wait, I just said what you said, nevermind, I skimmed too fast..

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> mod_cgi does no waiting. :) The only wait appears to be in the cleanup yes it does, but after mod_perl cleanups are run. see free_proc_chain() in alloc.c I also realized last night, you might want to look at Apache::Scoreboard, there's some cpu info in there that might be handy for your mod

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> "Doug" == Doug MacEachern <[EMAIL PROTECTED]> writes: >> My CPU-based limiter is working quite nicely. It lets oodles of >> static pages be served, but if someone starts doing CPU intensive >> stuff, they get booted for hogging my server machine. The nice thing >> is that I return a stand

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> My CPU-based limiter is working quite nicely. It lets oodles of > static pages be served, but if someone starts doing CPU intensive > stuff, they get booted for hogging my server machine. The nice thing > is that I return a standard "503" error including a "retry-after", so > if it is a legiti

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

On Wed, Nov 24, 1999 at 07:48:24AM -0800, Randal L. Schwartz wrote: > > my @FIELDS = > qw(when host method url user referer browser status bytes > wall cpuuser cpusys cpucuser cpucsys); > my $INSERT = > "INSERT INTO $DB_TABLE (". > (join ",", @FIELDS). > "

Re: embperl pages and braindead sucking robots

On Wed, Nov 24, 1999 at 05:26:08AM +0100, Gerald Richter wrote: > > I have for the first time encountered the problem that some braindead > > web robot (ExtractorPro) attempted to download all of the site and > > appended some random URL segment at the end of an embedded perl page. I > > use suffi

Re: Limiting CPU (was Re: embperl pages and braindead sucking robots)

> "Barry" == Barry Robison <[EMAIL PROTECTED]> writes: Barry> On Wed, Nov 24, 1999 at 07:31:36AM -0800, Randal L. Schwartz wrote: >> >> I also added a DBILogger that logs CPU times, so I can see which pages >> on my system are burning the most CPU, and even tell which hosts suck >> down the

Limiting CPU (was Re: embperl pages and braindead sucking robots)

> "Gerald" == Gerald Richter <[EMAIL PROTECTED]> writes: Gerald> Maybe Apache::SpeedLimit is helpfull. It limits the number of pages one Gerald> client can fetch in per time. There a other Apache modules to block robots, Gerald> look at the Apache module list. My CPU-based limiter is working

RE: embperl pages and braindead sucking robots

> I have for the first time encountered the problem that some braindead > web robot (ExtractorPro) attempted to download all of the site and > appended some random URL segment at the end of an embedded perl page. I > use suffix .phtml for these pages, and the url looked like >

RE: embperl pages and braindead sucking robots

Jens-Uwe Mager wrote: > I have for the first time encountered the problem that some braindead > web robot (ExtractorPro) attempted to download all of the site and > appended some random URL segment at the end of an embedded > perl page. I > use suffix .phtml for these pages, and the url looked l

embperl pages and braindead sucking robots

I have for the first time encountered the problem that some braindead web robot (ExtractorPro) attempted to download all of the site and appended some random URL segment at the end of an embedded perl page. I use suffix .phtml for these pages, and the url looked like