Hi Claudio,
On Wed, Aug 09, 2017 at 10:09:47AM +0200, Claudio Kuenzler wrote:
> Hi Willy,
>
>
> > Now the question is, does it cause any problem for you or is it just that
> > it came as a surprize and you were worried that it could cause problems ?
> >
>
> Yes, unfortunately it does create a problem.
>
> Each backend server (a SOAP API) can only handle up to 7 concurrent
> connections. The 8th connection freezes/is waiting for a server response.
> In order to satisfy this max connection requirement to the backend
> server(s) I wanted to use "maxconn 6".
> My hope was that HAProxy would not allow more than 6 concurrent connections
> going to each backend server.
> From POV of the SOAP API, an additional connection (7) is added because of
> the regular healthcheck from HAProxy.
OK.
> So now we have the problem that idle connections are not accounted for and
> HAProxy keeps letting new connections going through.
> This causes the SOAP backend servers to freeze up.
I see. To be honnest, this is the first ever such report over the last
5 years that server-side keep-alive was implemented, so this tends to
confirm that this server is not behaving like most others.
> > The possible alternative would be to have an option to say that idle
> > connections are accounted for and that some of them will be killed before
> > passing a new connection to the server, but that will significantly reduce
> > the efficiency of server-side keep-alive.
> >
>
> Yes, such an option would be really helpful. Should probably be turned off
> by default, but it would be a great help for such scenarios.
> But none of them should be killed if new requests are passed over to the
> backend server.
Yes we need to kill them, otherwise you'll end up exactly in the current
situation, except that instead of having the extra connection queued at
the server and waiting there for an idle connection to terminate, it would
be queued into haproxy waiting for such an idle connection to terminate.
> In fact the option could do the same as counting the number of established
> tcp connections going through HAProxy to the backend server and handle this
> as CUR value.
I see but basically you'll never be able to send requests there due to
the pending (unused) idle connections blocking the count to a high value.
> The maxconn setting is then reached by all connections (whether they're
> active or idle) resulting in HAProxy returning a 503 error.
This is becoming a bit ugly.
> > If you're really short on server-side connections and want to optimize
> > them as much as possible, you can try to enable "http-reuse".
>
> That's a good idea, but unfortunately won't work in this case.
> Each session, even idle sessions, have a bound "ticket" in the SOAP API. A
> reuse would basically hijack a session already in progress resulting in
> data corruption.
So in fact this "application" pretends to speak HTTP but is not compatible
with the HTTP spec. You're possibly taking risks by placing HTTP components
between the client and this thing.
> I know it's a weird application design and I personally haven't ever seen
> anything like this before.
I easily believe you ;-)
> I'm happy to provide my time/support for you to get in place such an
> option. Just let me know.
Anyway it will not happen before 1.8 is released and we start to work on
1.9, as it would take time away from the already planned features.
> In the meantime I'll probably try to solve this using iptables limits
> behind HAProxy (between HAProxy and backend server).
It won't really work. There might be something which can work, which is
to chain to a TCP listener. It will enforce the maxconn count at the TCP
level. By having this :
listen foo
mode http
...
server app02-1 127.0.0.1:10001
server app02-2 127.0.0.1:10002
server app02-3 127.0.0.1:10003
listen app02-1
mode tcp
bind 127.0.0.1:10001
server app02-1 10.10.10.12:12345 maxconn 6 maxqueue 0
listen app02-2
mode tcp
bind 127.0.0.1:10002
server app02-1 10.10.10.12:12346 maxconn 6 maxqueue 0
...
By the way, looking at your config, I'm now wondering why
you are using HTTP mode in your frontend/backend instead of
TCP mode. You're adding a cookie but you mentionned that the
application doesn't support requests being mixed over connections
so I guess that the stickiness is only ensured at the connection
level and the cookie very likely is useless. Then by using only
"mode tcp" you could have exactly what you need, ie: maxconn
enforced at the TCP level.
Regards,
willy
