Re: fullconn not working
*From:* Jerome Magnin [mailto:jmag...@haproxy.com]
*Sent:* Tuesday, July 16, 2019, 10:19 EDT
*To:* Patrick Hemmer
*Cc:* Pavlos Parissis , haproxy@formilux.org
*Subject:* fullconn not working
Hi Patrick,
On Tue, Jul 16, 2019 at 09:40:31AM -0400, Patrick Hemmer wrote:
*From:* Pavlos Parissis [mailto:pavlos.paris...@gmail.com]
*Sent:* Tuesday, July 16, 2019, 09:32 EDT
*To:* haproxy@formilux.org
*Cc:* Patrick Hemmer
*Subject:* fullconn not working
On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote:
I'm trying to get fullconn working, and can't seem to do so. I dunno if
it's a bug, or if it's my understanding that's wrong.
Basically my goal is to prevent the cumulative total of all connections
to all servers in a pool from exceeding a certain value.
For example I might have 10 servers, each with a maxconn of 10. But I
want to configure haproxy with a pool-wide limit of 50, so that even if
the connections are well distributed and no one server is maxed out,
after 50 connections to all servers, haproxy will still start to queue
instead.
fullconn seems like the right way to accomplish this, however I cannot
get it to work. I've tried a simple setup of 2 servers, each with
`maxconn 3`, and then a backend `fullconn 2`, which should result in
queuing after 2 simultaneous connections, however it doesn't. If I send
4 connections, all 4 are simultaneously sent to the backend servers.
Here's my test config:
defaults
log 127.0.0.1:1234 daemon
mode http
option httplog
timeout queue 5s
frontend f1
bind :8001
default_backend b1
backend b1
fullconn 2
server s1 127.0.0.1:8081 minconn 1 maxconn 3
server s2 127.0.0.1:8081 minconn 1 maxconn 3
Here's how I test:
for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done
And here's the logs:
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"
Your e-mail client mangled above log lines and as a result they are bit
unreadable.
The 4th field from `4/4/3/2/0` is srv_conn *2* which is below the maxconn of
*3*, so haproxy did
the right thing as it didn't allow more than *full_conn* connections to be
concurrently opened
against the server.
Cheers,
Pavlos
maxconn and fullconn are different settings.
maxconn:
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#maxconn%20(Server%20and%20default-server%20options)
fullconn:
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#fullconn
-Patrick
fullconn is not used to set a limit on the amount of connections handled by the
backend. It is used to have a 'dynamic' maxconn value on server lines.
this dymanic maxconn value will never be below minconn, and never higher than
maxconn. When the amount of connections handled by backend is between 0 and
fullconn, this dynamic maxconn value is somewhere between minconn and maxconn,
proportionnate to where we are between 0 and fullconn. When fullconn is reached,
dynamic maxconn equals the maxconn you set on server line.
With the values you set, when you reach 2 connections at backend level, servers
will allow 3 at most, each.
Jérôme
Thanks, I think I'm following now. My understanding was backwards.
I'm guessing the use case for this is so that you have maxconn at a
value where servers have a good response time. But then if you start
queueing too much, then maxconn is raised, trading slower response time
for higher throughput.
So my next question, is there any way to set a backend connection limit
(after which connections get queued)? I want to limit the number of
active connections across the whole pool, regardless of the number of
servers, or what the limit on each server is.
The reason is that there are shared resources (e.g. a database, NAS
filesystem, etc) behind a pool, and that shared resource can only handle
so much load. We can divide the cumulative limit across the number of
servers in the pool, but if some of the servers are out of the pool,
then that limit shrinks and we're not utilizing the full capacity of the
shared resources.
-Patrick
Re: fullconn not working
Hi Patrick,
On Tue, Jul 16, 2019 at 09:40:31AM -0400, Patrick Hemmer wrote:
>
>
>
> *From:* Pavlos Parissis [mailto:pavlos.paris...@gmail.com]
> *Sent:* Tuesday, July 16, 2019, 09:32 EDT
> *To:* haproxy@formilux.org
> *Cc:* Patrick Hemmer
> *Subject:* fullconn not working
>
> > On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote:
> > > I'm trying to get fullconn working, and can't seem to do so. I dunno if
> > > it's a bug, or if it's my understanding that's wrong.
> > > Basically my goal is to prevent the cumulative total of all connections
> > > to all servers in a pool from exceeding a certain value.
> > > For example I might have 10 servers, each with a maxconn of 10. But I
> > > want to configure haproxy with a pool-wide limit of 50, so that even if
> > > the connections are well distributed and no one server is maxed out,
> > > after 50 connections to all servers, haproxy will still start to queue
> > > instead.
> > >
> > > fullconn seems like the right way to accomplish this, however I cannot
> > > get it to work. I've tried a simple setup of 2 servers, each with
> > > `maxconn 3`, and then a backend `fullconn 2`, which should result in
> > > queuing after 2 simultaneous connections, however it doesn't. If I send
> > > 4 connections, all 4 are simultaneously sent to the backend servers.
> > >
> > > Here's my test config:
> > > defaults
> > > log 127.0.0.1:1234 daemon
> > > mode http
> > > option httplog
> > > timeout queue 5s
> > > frontend f1
> > > bind :8001
> > > default_backend b1
> > > backend b1
> > > fullconn 2
> > > server s1 127.0.0.1:8081 minconn 1 maxconn 3
> > > server s2 127.0.0.1:8081 minconn 1 maxconn 3
> > >
> > > Here's how I test:
> > > for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done
> > >
> > > And here's the logs:
> > > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119
> > > [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
> > > 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
> > > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117
> > > [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
> > > 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
> > > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118
> > > [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
> > > 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
> > > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120
> > > [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
> > > 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"
> > >
> > >
> > Your e-mail client mangled above log lines and as a result they are bit
> > unreadable.
> > The 4th field from `4/4/3/2/0` is srv_conn *2* which is below the maxconn
> > of *3*, so haproxy did
> > the right thing as it didn't allow more than *full_conn* connections to be
> > concurrently opened
> > against the server.
> >
> > Cheers,
> > Pavlos
> maxconn and fullconn are different settings.
> maxconn:
> https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#maxconn%20(Server%20and%20default-server%20options)
> fullconn:
> https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#fullconn
>
> -Patrick
fullconn is not used to set a limit on the amount of connections handled by the
backend. It is used to have a 'dynamic' maxconn value on server lines.
this dymanic maxconn value will never be below minconn, and never higher than
maxconn. When the amount of connections handled by backend is between 0 and
fullconn, this dynamic maxconn value is somewhere between minconn and maxconn,
proportionnate to where we are between 0 and fullconn. When fullconn is reached,
dynamic maxconn equals the maxconn you set on server line.
With the values you set, when you reach 2 connections at backend level, servers
will allow 3 at most, each.
Jérôme
Re: fullconn not working
*From:* Pavlos Parissis [mailto:pavlos.paris...@gmail.com]
*Sent:* Tuesday, July 16, 2019, 09:32 EDT
*To:* haproxy@formilux.org
*Cc:* Patrick Hemmer
*Subject:* fullconn not working
On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote:
I'm trying to get fullconn working, and can't seem to do so. I dunno if
it's a bug, or if it's my understanding that's wrong.
Basically my goal is to prevent the cumulative total of all connections
to all servers in a pool from exceeding a certain value.
For example I might have 10 servers, each with a maxconn of 10. But I
want to configure haproxy with a pool-wide limit of 50, so that even if
the connections are well distributed and no one server is maxed out,
after 50 connections to all servers, haproxy will still start to queue
instead.
fullconn seems like the right way to accomplish this, however I cannot
get it to work. I've tried a simple setup of 2 servers, each with
`maxconn 3`, and then a backend `fullconn 2`, which should result in
queuing after 2 simultaneous connections, however it doesn't. If I send
4 connections, all 4 are simultaneously sent to the backend servers.
Here's my test config:
defaults
log 127.0.0.1:1234 daemon
mode http
option httplog
timeout queue 5s
frontend f1
bind :8001
default_backend b1
backend b1
fullconn 2
server s1 127.0.0.1:8081 minconn 1 maxconn 3
server s2 127.0.0.1:8081 minconn 1 maxconn 3
Here's how I test:
for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done
And here's the logs:
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"
Your e-mail client mangled above log lines and as a result they are bit
unreadable.
The 4th field from `4/4/3/2/0` is srv_conn *2* which is below the maxconn of
*3*, so haproxy did
the right thing as it didn't allow more than *full_conn* connections to be
concurrently opened
against the server.
Cheers,
Pavlos
maxconn and fullconn are different settings.
maxconn:
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#maxconn%20(Server%20and%20default-server%20options)
fullconn:
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#fullconn
-Patrick
Re: fullconn not working
On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote:
> I'm trying to get fullconn working, and can't seem to do so. I dunno if
> it's a bug, or if it's my understanding that's wrong.
> Basically my goal is to prevent the cumulative total of all connections
> to all servers in a pool from exceeding a certain value.
> For example I might have 10 servers, each with a maxconn of 10. But I
> want to configure haproxy with a pool-wide limit of 50, so that even if
> the connections are well distributed and no one server is maxed out,
> after 50 connections to all servers, haproxy will still start to queue
> instead.
>
> fullconn seems like the right way to accomplish this, however I cannot
> get it to work. I've tried a simple setup of 2 servers, each with
> `maxconn 3`, and then a backend `fullconn 2`, which should result in
> queuing after 2 simultaneous connections, however it doesn't. If I send
> 4 connections, all 4 are simultaneously sent to the backend servers.
>
> Here's my test config:
> defaults
> log 127.0.0.1:1234 daemon
> mode http
> option httplog
> timeout queue 5s
> frontend f1
> bind :8001
> default_backend b1
> backend b1
> fullconn 2
> server s1 127.0.0.1:8081 minconn 1 maxconn 3
> server s2 127.0.0.1:8081 minconn 1 maxconn 3
>
> Here's how I test:
> for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done
>
> And here's the logs:
> <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119
> [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
> 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
> <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117
> [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
> 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
> <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118
> [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
> 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
> <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120
> [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
> 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"
>
>
Your e-mail client mangled above log lines and as a result they are bit
unreadable.
The 4th field from `4/4/3/2/0` is srv_conn *2* which is below the maxconn of
*3*, so haproxy did
the right thing as it didn't allow more than *full_conn* connections to be
concurrently opened
against the server.
Cheers,
Pavlos
Re: fullconn not working
*From:* Patrick Hemmer [mailto:hapr...@stormcloud9.net]
*Sent:* Friday, June 28, 2019, 11:50 EDT
*To:* HAProxy
*Subject:* fullconn not working
I'm trying to get fullconn working, and can't seem to do so. I dunno
if it's a bug, or if it's my understanding that's wrong.
Basically my goal is to prevent the cumulative total of all
connections to all servers in a pool from exceeding a certain value.
For example I might have 10 servers, each with a maxconn of 10. But I
want to configure haproxy with a pool-wide limit of 50, so that even
if the connections are well distributed and no one server is maxed
out, after 50 connections to all servers, haproxy will still start to
queue instead.
fullconn seems like the right way to accomplish this, however I cannot
get it to work. I've tried a simple setup of 2 servers, each with
`maxconn 3`, and then a backend `fullconn 2`, which should result in
queuing after 2 simultaneous connections, however it doesn't. If I
send 4 connections, all 4 are simultaneously sent to the backend servers.
Here's my test config:
defaults
log 127.0.0.1:1234 daemon
mode http
option httplog
timeout queue 5s
frontend f1
bind :8001
default_backend b1
backend b1
fullconn 2
server s1 127.0.0.1:8081 minconn 1 maxconn 3
server s2 127.0.0.1:8081 minconn 1 maxconn 3
Here's how I test:
for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done
And here's the logs:
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -
4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -
4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"
So am I misunderstanding how fullconn works? Or is there a bug?
I've tested with 2.0.1, 1.9.8, and 1.8.13.
-Patrick
Anyone with any thoughts on this? Anyone using fullconn successfully?
As an additional item of note that makes me suspect this is a bug even
more is that when viewing the stats page, it shows the number of
sessions is greater than the limit.
-Patrick
