Re: maxconn not respecting idle connections?
Hello, Am 09.08.2017 um 11:12 schrieb Willy Tarreau: > > There might be something which can work, which is > to chain to a TCP listener. It will enforce the maxconn count at the TCP > level. Or a simpler workaround, disable http keepalive on the backend with "option http-server-close". cheers, lukas
Re: maxconn not respecting idle connections?
Salut Willy, > So in fact this "application" pretends to speak HTTP but is not compatible > with the HTTP spec. You're possibly taking risks by placing HTTP components > between the client and this thing. > Yeah, tell that Adobe. The application we're talking about are Indesign Server instances. > > I'm happy to provide my time/support for you to get in place such an > > option. Just let me know. > > Anyway it will not happen before 1.8 is released and we start to work on > 1.9, as it would take time away from the already planned features. > OK whenever the timing's ready, let me know. > > > In the meantime I'll probably try to solve this using iptables limits > > behind HAProxy (between HAProxy and backend server). > > It won't really work. It does. I've set a REJECT with a connection number above 7 like this (for each backend server obvisously): iptables -A OUTPUT -p tcp --syn -d 10.10.10.12/32 --dport 18390 -m connlimit --connlimit-above 7 -j REJECT --reject-with tcp-reset The only downside I see with this setup is that the healthcheck from HAProxy returns a L4TOUT when 7 connections are already in use by the (application-)client. The 8th connection (healthcheck coming from HAProxy) is rejected by iptables. I'm aware that I'm working around the issue here, but so far this is the closest solution I came up with to meet the application requirements. > There might be something which can work, which is > to chain to a TCP listener. It will enforce the maxconn count at the TCP > level. By having this : > > listen foo > mode http > ... > server app02-1 127.0.0.1:10001 > server app02-2 127.0.0.1:10002 > server app02-3 127.0.0.1:10003 > > listen app02-1 > mode tcp > bind 127.0.0.1:10001 > server app02-1 10.10.10.12:12345 maxconn 6 maxqueue 0 > > listen app02-2 > mode tcp > bind 127.0.0.1:10002 > server app02-1 10.10.10.12:12346 maxconn 6 maxqueue 0 > > ... > > By the way, looking at your config, I'm now wondering why > you are using HTTP mode in your frontend/backend instead of > TCP mode. You're adding a cookie but you mentionned that the > application doesn't support requests being mixed over connections > so I guess that the stickiness is only ensured at the connection > level and the cookie very likely is useless. Then by using only > "mode tcp" you could have exactly what you need, ie: maxconn > enforced at the TCP level. > This is a very good idea, but I don't think it would work. I wrote in another mailing list thread before, that once a connection was send to a backend server, all future requests MUST go to the same backend server again (no failover, no balancing - a stale and fixed connection between client and backend server). To my current knowledge, this is only possible by setting a cookie which can only be read in http mode. And "balance source" in this case won't help, because the source IP of the client will be the same. If I'm mistaken, please let me know. In your tcp example I just noticed "maxqueue 0". Does this actually disable the queue? I asked this in my previous mailing list thread :) cheers, ck
Re: maxconn not respecting idle connections?
Hi Claudio, On Wed, Aug 09, 2017 at 10:09:47AM +0200, Claudio Kuenzler wrote: > Hi Willy, > > > > Now the question is, does it cause any problem for you or is it just that > > it came as a surprize and you were worried that it could cause problems ? > > > > Yes, unfortunately it does create a problem. > > Each backend server (a SOAP API) can only handle up to 7 concurrent > connections. The 8th connection freezes/is waiting for a server response. > In order to satisfy this max connection requirement to the backend > server(s) I wanted to use "maxconn 6". > My hope was that HAProxy would not allow more than 6 concurrent connections > going to each backend server. > From POV of the SOAP API, an additional connection (7) is added because of > the regular healthcheck from HAProxy. OK. > So now we have the problem that idle connections are not accounted for and > HAProxy keeps letting new connections going through. > This causes the SOAP backend servers to freeze up. I see. To be honnest, this is the first ever such report over the last 5 years that server-side keep-alive was implemented, so this tends to confirm that this server is not behaving like most others. > > The possible alternative would be to have an option to say that idle > > connections are accounted for and that some of them will be killed before > > passing a new connection to the server, but that will significantly reduce > > the efficiency of server-side keep-alive. > > > > Yes, such an option would be really helpful. Should probably be turned off > by default, but it would be a great help for such scenarios. > But none of them should be killed if new requests are passed over to the > backend server. Yes we need to kill them, otherwise you'll end up exactly in the current situation, except that instead of having the extra connection queued at the server and waiting there for an idle connection to terminate, it would be queued into haproxy waiting for such an idle connection to terminate. > In fact the option could do the same as counting the number of established > tcp connections going through HAProxy to the backend server and handle this > as CUR value. I see but basically you'll never be able to send requests there due to the pending (unused) idle connections blocking the count to a high value. > The maxconn setting is then reached by all connections (whether they're > active or idle) resulting in HAProxy returning a 503 error. This is becoming a bit ugly. > > If you're really short on server-side connections and want to optimize > > them as much as possible, you can try to enable "http-reuse". > > That's a good idea, but unfortunately won't work in this case. > Each session, even idle sessions, have a bound "ticket" in the SOAP API. A > reuse would basically hijack a session already in progress resulting in > data corruption. So in fact this "application" pretends to speak HTTP but is not compatible with the HTTP spec. You're possibly taking risks by placing HTTP components between the client and this thing. > I know it's a weird application design and I personally haven't ever seen > anything like this before. I easily believe you ;-) > I'm happy to provide my time/support for you to get in place such an > option. Just let me know. Anyway it will not happen before 1.8 is released and we start to work on 1.9, as it would take time away from the already planned features. > In the meantime I'll probably try to solve this using iptables limits > behind HAProxy (between HAProxy and backend server). It won't really work. There might be something which can work, which is to chain to a TCP listener. It will enforce the maxconn count at the TCP level. By having this : listen foo mode http ... server app02-1 127.0.0.1:10001 server app02-2 127.0.0.1:10002 server app02-3 127.0.0.1:10003 listen app02-1 mode tcp bind 127.0.0.1:10001 server app02-1 10.10.10.12:12345 maxconn 6 maxqueue 0 listen app02-2 mode tcp bind 127.0.0.1:10002 server app02-1 10.10.10.12:12346 maxconn 6 maxqueue 0 ... By the way, looking at your config, I'm now wondering why you are using HTTP mode in your frontend/backend instead of TCP mode. You're adding a cookie but you mentionned that the application doesn't support requests being mixed over connections so I guess that the stickiness is only ensured at the connection level and the cookie very likely is useless. Then by using only "mode tcp" you could have exactly what you need, ie: maxconn enforced at the TCP level. Regards, willy
Re: maxconn not respecting idle connections?
Hi Willy, > Now the question is, does it cause any problem for you or is it just that > it came as a surprize and you were worried that it could cause problems ? > Yes, unfortunately it does create a problem. Each backend server (a SOAP API) can only handle up to 7 concurrent connections. The 8th connection freezes/is waiting for a server response. In order to satisfy this max connection requirement to the backend server(s) I wanted to use "maxconn 6". My hope was that HAProxy would not allow more than 6 concurrent connections going to each backend server. >From POV of the SOAP API, an additional connection (7) is added because of the regular healthcheck from HAProxy. So now we have the problem that idle connections are not accounted for and HAProxy keeps letting new connections going through. This causes the SOAP backend servers to freeze up. > The possible alternative would be to have an option to say that idle > connections are accounted for and that some of them will be killed before > passing a new connection to the server, but that will significantly reduce > the efficiency of server-side keep-alive. > Yes, such an option would be really helpful. Should probably be turned off by default, but it would be a great help for such scenarios. But none of them should be killed if new requests are passed over to the backend server. In fact the option could do the same as counting the number of established tcp connections going through HAProxy to the backend server and handle this as CUR value. The maxconn setting is then reached by all connections (whether they're active or idle) resulting in HAProxy returning a 503 error. > > If you're really short on server-side connections and want to optimize > them as much as possible, you can try to enable "http-reuse". That's a good idea, but unfortunately won't work in this case. Each session, even idle sessions, have a bound "ticket" in the SOAP API. A reuse would basically hijack a session already in progress resulting in data corruption. I know it's a weird application design and I personally haven't ever seen anything like this before. I'm happy to provide my time/support for you to get in place such an option. Just let me know. In the meantime I'll probably try to solve this using iptables limits behind HAProxy (between HAProxy and backend server).
Re: maxconn not respecting idle connections?
Hi Claudio, On Tue, Aug 08, 2017 at 07:16:17AM +0200, Claudio Kuenzler wrote: > Hi, > > I've set "hard limits" with maxconn for each backend server but it seems > that established (keep-alive) connections are not accounted for in the > stats. This leads to HAProxy allowing more connections to the backend > server than actually defined with the maxconn value. (...) > To me it looks like only connections with current data-transfer are > accounted for in the CUR statistics, idle connections somehow drop out and > HAProxy then allows additional connections to the backend, causing it to > stale (it cannot handle more than 7 sessions). Am I wrong? Did I > misinterpret the maxconn purpose? No you're totally right and it's by design. The thing is, you don't want to leave requests waiting in a server's queue while the server has a ton of idle connections. The vast majority of servers nowadays have a dispatching frontend which is mostly insensitive to idle connections, and only really see outstanding requests. This has been even more true since all browsers started to implement the pre-connect feature a few years ago, establishing idle connections to sites you've recently visited just in case you'd want to visit them again, resulting in a huge amount of idle connections on servers. So when using server-side keep-alive we continue to ensure that the server never has to process more than a given number of outstanding requests, and idle connections are not accounted for. Now the question is, does it cause any problem for you or is it just that it came as a surprize and you were worried that it could cause problems ? The possible alternative would be to have an option to say that idle connections are accounted for and that some of them will be killed before passing a new connection to the server, but that will significantly reduce the efficiency of server-side keep-alive. If you're really short on server-side connections and want to optimize them as much as possible, you can try to enable "http-reuse". It will allow sharing of idle connections between frontend connections so that a request may be sent over an existing connection. It is the way to achieve the lowest number of concurrent connections on the server side. But not all applications support this (most do nowadays), you need to check (eg: some try to retrieve the source address once per connection for logging purposes for example). Regards, Willy
