Re: Small patch for the appsession feature

2009-11-16 Thread Aleksandar Lazic
On Mon 16.11.2009 08:52, Willy Tarreau wrote: Hi, On Sun, Nov 15, 2009 at 10:28:21PM +0100, Aleksandar Lazic wrote: Hi Cyril, On Fre 13.11.2009 22:50, Cyril Bonté wrote: Hello Willy, [snipp] First I added it as I did for request-learn but shouldn't it be better to define these options

Re: Small patch for the appsession feature

2009-11-16 Thread Willy Tarreau
On Mon, Nov 16, 2009 at 10:08:56AM +0100, Aleksandar Lazic wrote: What you can do however is to create a new prefix keyword like we have for timeout or tcp-request and put the flags somewhere else. appsession would have been fine but it's already used. Maybe you can use appcookie ? Something

Re: Preventing bots from starving other users?

2009-11-16 Thread Wout Mertens
Hi John, On Nov 15, 2009, at 8:29 PM, John Lauro wrote: I would probably do that sort of throttling at the OS level with iptables, etc... Hmmm How? I don't want to throw away the requests, just queue them. Looking for iptables rate limiting it seems that you can only drop the request.

Re: Small patch for the appsession feature

2009-11-16 Thread Willy Tarreau
On Mon, Nov 16, 2009 at 01:04:17PM +0100, Aleksandar Lazic wrote: (...) app* delimiterstart (default ;) app* delimiterstop (defautl =) I'm sorry, I don't understand what you mean. Could you give an example of configuration using this ? For example: if in the url the delimiter changes

Re: Preventing bots from starving other users?

2009-11-16 Thread Karsten Elfenbein
Just create an additional backend and assign the bots to it. You can set queues and max connections there as needed. Also an additional tip might be to adjust the robots.txt file as some bots can be slowed down. http://www.google.com/support/webmasters/bin/answer.py?answer=48620 Check if the

RE: Preventing bots from starving other users?

2009-11-16 Thread John Lauro
Oopps, my bad... It's actually tc and not iptables. Googletc qdisc for some info. You could allow your local ips go unrestricted, and throttle all other IPs to 512kb/sec for example. What software is the running on? I assume it's not running under apache or there would be some ways to

RE: Preventing bots from starving other users?

2009-11-16 Thread John Marrett
You can ask (polite) bots to throttle their request rates and simultaneous requests. It think that you'd probably be quite interested in the crawl-delay directive: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_direc tive This is respected by at least MSN and Yahoo.

Re: Preventing bots from starving other users?

2009-11-16 Thread German Gutierrez
Perhaps this plugin could be useful, never used, tho: http://twiki.org/cgi-bin/view/Plugins.TWikiCacheAddOn On Mon, Nov 16, 2009 at 11:46 AM, Wout Mertens wout.mert...@gmail.comwrote: On Nov 16, 2009, at 1:47 PM, Karsten Elfenbein wrote: Just create an additional backend and assign the bots

Re: Preventing bots from starving other users?

2009-11-16 Thread Wout Mertens
Schweet! I'll give that a shot. Wout. On Nov 16, 2009, at 4:08 PM, Karsten Elfenbein wrote: you can just create the backend in haproxy and use the same backend server definition no need to reconfigure apache put like 7 max sessions for normal users on one backend and 2 for maxsessions

Re: Small patch for the appsession feature

2009-11-16 Thread Cyril Bonté
Le lundi 16 novembre 2009 13:41:14, Willy Tarreau a écrit : On Mon, Nov 16, 2009 at 01:04:17PM +0100, Aleksandar Lazic wrote: (...) app* delimiterstart (default ;) app* delimiterstop (defautl =) I'm sorry, I don't understand what you mean. Could you give an example of configuration