On Mon 16.11.2009 08:52, Willy Tarreau wrote:
Hi,
On Sun, Nov 15, 2009 at 10:28:21PM +0100, Aleksandar Lazic wrote:
Hi Cyril,
On Fre 13.11.2009 22:50, Cyril Bonté wrote:
Hello Willy,
[snipp]
First I added it as I did for request-learn but shouldn't it be
better to define these options
On Mon, Nov 16, 2009 at 10:08:56AM +0100, Aleksandar Lazic wrote:
What you can do however is to create a new prefix keyword like we have
for timeout or tcp-request and put the flags somewhere
else. appsession would have been fine but it's already used. Maybe
you can use appcookie ? Something
Hi John,
On Nov 15, 2009, at 8:29 PM, John Lauro wrote:
I would probably do that sort of throttling at the OS level with iptables,
etc...
Hmmm How? I don't want to throw away the requests, just queue them. Looking
for iptables rate limiting it seems that you can only drop the request.
On Mon, Nov 16, 2009 at 01:04:17PM +0100, Aleksandar Lazic wrote:
(...)
app* delimiterstart (default ;)
app* delimiterstop (defautl =)
I'm sorry, I don't understand what you mean. Could you give an example
of configuration using this ?
For example:
if in the url the delimiter changes
Just create an additional backend and assign the bots to it.
You can set queues and max connections there as needed.
Also an additional tip might be to adjust the robots.txt file as some bots can
be slowed down.
http://www.google.com/support/webmasters/bin/answer.py?answer=48620
Check if the
Oopps, my bad... It's actually tc and not iptables. Googletc qdisc
for some info.
You could allow your local ips go unrestricted, and throttle all other IPs
to 512kb/sec for example.
What software is the running on? I assume it's not running under apache or
there would be some ways to
You can ask (polite) bots to throttle their request rates and
simultaneous requests. It think that you'd probably be quite interested
in the crawl-delay directive:
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_direc
tive
This is respected by at least MSN and Yahoo.
Perhaps this plugin could be useful, never used, tho:
http://twiki.org/cgi-bin/view/Plugins.TWikiCacheAddOn
On Mon, Nov 16, 2009 at 11:46 AM, Wout Mertens wout.mert...@gmail.comwrote:
On Nov 16, 2009, at 1:47 PM, Karsten Elfenbein wrote:
Just create an additional backend and assign the bots
8 matches
Mail list logo
