[PATCH] BUG/MINOR: fix maxaccept computation according to the frontend process range

[Cyril Bonté]

commit 7c0ffd23 is only considering the explicit use of the "process" keyword
on the listeners. But at this step, if it's not defined in the configuration,
the listener bind_proc mask is set to 0. As a result, the code will compute
the maxaccept value based on only 1 process, which is not always true.

For example :
  global
nbproc 4

  frontend test
bind-process 1-2
bind :80

Here, the maxaccept value for the "test" frontend was set to the global
tune.maxaccept value (default to 64), whereas it should consider 2 processes
will accept connections. As of the documentation, the value should be divided
by twice the number of processes the listener is bound to.

To fix this, we can consider that if no mask is set to the listener, we take
the frontend mask.

This is not critical but it can introduce unfairness distribution of the
incoming connections across the processes.

It should be backported to the same branches as commit 7c0ffd23 (1.6 and 1.5
were in the scope).
---
 src/cfgparse.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/cfgparse.c b/src/cfgparse.c
index c3b29d4..2400559 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -8741,7 +8741,7 @@ out_uri_auth_compat:
int nbproc;
 
nbproc = my_popcountl(curproxy->bind_proc &
- listener->bind_conf->bind_proc &
+ (listener->bind_conf->bind_proc ? 
listener->bind_conf->bind_proc : curproxy->bind_proc) &
  nbits(global.nbproc));
 
if (!nbproc) /* no intersection between listener and 
frontend */
-- 
2.8.0.rc3

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[Willy Tarreau]

On Thu, Apr 14, 2016 at 02:22:36PM +0200, Janusz Dziemidowicz wrote:
> 2016-04-14 12:05 GMT+02:00 Willy Tarreau :
> > Hi David,
> >
> > On Wed, Apr 13, 2016 at 03:19:45PM -0500, David Martin wrote:
> >> This is my first attempt at a patch, I'd love to get some feedback on this.
> >>
> >> Adds support for SSL_CTX_set_ecdh_auto which is available in OpenSSL 1.0.2.
> >
> >> From 05bee3e95e5969294998fb9e2794ef65ce5a6c1f Mon Sep 17 00:00:00 2001
> >> From: David Martin 
> >> Date: Wed, 13 Apr 2016 15:09:35 -0500
> >> Subject: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection
> >>
> >> Use SSL_CTX_set_ecdh_auto if the OpenSSL version supports it, this
> >> allows the server to negotiate ECDH curves much like it does ciphers.
> >> Prefered curves can be specified using the existing ecdhe bind options
> >> (ecdhe secp384r1:prime256v1)
> >
> > Could it have a performance impact ? I mean, may this allow a client to
> > force the server to use curves that imply harder computations for example ?
> > I'm asking because some people got seriously hit by the move from dhparm
> > 1024 to 2048, so if this can come with a performance impact we possibly want
> > to let the user configure it.
> 
> Switching ECDHE curves can have performance impact, for example result
> of openssl speed on my laptop:
>  256 bit ecdh (nistp256)   0.0003s   2935.3
>  384 bit ecdh (nistp384)   0.0027s364.9
>  521 bit ecdh (nistp521)   0.0016s623.2
> The difference is so high for nistp256 because OpenSSL has heavily
> optimized implementation
> (https://www.imperialviolet.org/2010/12/04/ecc.html).

Wow, and despite this you want to let the client force the server to
switch to 384 ? Looks like a hue DoS to me.

> Apart from calling SSL_CTX_set_ecdh_auto() this patch also takes into
> account user supplied curve list, so users can customize this as
> needed (currently haproxy only allows to select one curve, which is a
> limitation of older OpenSSL versions).

OK.

> However, this patch reuses bind option 'ecdhe'. Currently it is
> documented to accept only one curve. I believe it should be at least
> updated to state that multiple curves can be used with sufficiently
> new OpenSSL.

That makes sense.

> Also, I'm not sure what will happen when SSL_CTX_set1_curves_list() is
> called with NULL (no ecdhe bind option). Even if it is accepted by
> OpenSSL it will silently change haproxy default, before this patch it
> was only prime256v1 (as defined in ECDHE_DEFAULT_CURVE), afterward it
> will default to all curves supported by OpenSSL. Probably the best
> would be to keep current default, so it all works consistently in
> default configuration, regardless of version of haproxy and OpenSSL.

Agreed, we don't want it to change in the back of users. I'm not opposed
to adding more tuning options but we have to ensure that users will remain
safe after upgrading and will not be exposed to nistp384 without explicitly
asking for it.

Regards,
Willy

HAProxy 1.6, override for dns/NXdomains on parsing

[Michel Belleau]

Hi Baptiste.
(cc: HAProxy mailing-list)

I recently came across one of your posts from last year 
(http://permalink.gmane.org/gmane.comp.web.haproxy/22841) regarding how DNS 
records are resolved when loading new configuration values (either at parsing 
during initial startup, or on dynamic reconfiguration via the socket). In this 
post, you are referring to a possible enhancement:

Currently, HAProxy works like this: "init-addr libc,dns"
A new value could be "init-addr dns"
Or "init-addr 1.2.3.4,dns"

I believe none of this has been implemented yet, am I right? I am running into 
a situation I would like the latter — a condition can exist where my HAproxy 
would load before some of my backend server entries can be successfully 
resolved.

I tried something that seems to fit my particular need and I’d like to share 
with you and all — please see file-attach.

Essentially, this adds a new global option “override-nxdomain” that can be 
used, as its name implies, to override the return value when HAproxy uses the 
systems DNS resolution and receives the indication the hostname can’t be found. 
A possible use-case is to override (for example) with “localhost” or 
“127.0.0.1” directly so that HAproxy can initially start even if some records 
can’t be resolved yet. Once HAproxy starts to resolve on its own upon checking 
for health of the backend member, it will try to resolve again and hopefully 
get a valid answer at some point.

It seems to pass simple sanity/functionnal tests, but bear in mind this is my 
first submission to HAproxy and I would accept if it is considered a hack at 
this point. :-)



haproxy-1.6.4-nxdomain.diff
Description: haproxy-1.6.4-nxdomain.diff
 Thank you,Michel Belleau 

[no subject]

[Michel Belleau]

Ouotation

[info]

Dear Seller,My name is Ryan Williams from Tiestlin Ventures (We are Trading Company base in United States) we are interested on your products. Our company is looking for a reliable supplier who can provide a long term customer ship and maintain our customer’s specification items. 
Supplier could be from worldwide but quality should be good and the prices should be reasonable. Soonest reply with your quotation/ fob price will be highly appreciated for the further details of our specification order. I am waiting for a prompt response.Best Regards,Ryan WilliamsPurchasing Manager
Tiestlin Ventures CorporationAddress: 2013 Centre Road, Wilmington, Delaware, United States
Tel: +13028474903

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[David Martin]

Here's a revised patch, it throws a fatal config error if
SSL_CTX_set1_curves_list() fails.  The default echde option is used so
current configurations should not be impacted.

Sorry Janusz, forgot the list on my reply.

On Thu, Apr 14, 2016 at 10:37 AM, David Martin  wrote:
> Here's a revised patch, it throws a fatal config error if
> SSL_CTX_set1_curves_list() fails.  The default echde option is used so
> current configurations should not be impacted.
>
> On Thu, Apr 14, 2016 at 7:22 AM, Janusz Dziemidowicz
>  wrote:
>> 2016-04-14 12:05 GMT+02:00 Willy Tarreau :
>>> Hi David,
>>>
>>> On Wed, Apr 13, 2016 at 03:19:45PM -0500, David Martin wrote:
 This is my first attempt at a patch, I'd love to get some feedback on this.

 Adds support for SSL_CTX_set_ecdh_auto which is available in OpenSSL 1.0.2.
>>>
 From 05bee3e95e5969294998fb9e2794ef65ce5a6c1f Mon Sep 17 00:00:00 2001
 From: David Martin 
 Date: Wed, 13 Apr 2016 15:09:35 -0500
 Subject: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

 Use SSL_CTX_set_ecdh_auto if the OpenSSL version supports it, this
 allows the server to negotiate ECDH curves much like it does ciphers.
 Prefered curves can be specified using the existing ecdhe bind options
 (ecdhe secp384r1:prime256v1)
>>>
>>> Could it have a performance impact ? I mean, may this allow a client to
>>> force the server to use curves that imply harder computations for example ?
>>> I'm asking because some people got seriously hit by the move from dhparm
>>> 1024 to 2048, so if this can come with a performance impact we possibly want
>>> to let the user configure it.
>>
>> Switching ECDHE curves can have performance impact, for example result
>> of openssl speed on my laptop:
>>  256 bit ecdh (nistp256)   0.0003s   2935.3
>>  384 bit ecdh (nistp384)   0.0027s364.9
>>  521 bit ecdh (nistp521)   0.0016s623.2
>> The difference is so high for nistp256 because OpenSSL has heavily
>> optimized implementation
>> (https://www.imperialviolet.org/2010/12/04/ecc.html).
>>
>> Apart from calling SSL_CTX_set_ecdh_auto() this patch also takes into
>> account user supplied curve list, so users can customize this as
>> needed (currently haproxy only allows to select one curve, which is a
>> limitation of older OpenSSL versions).
>>
>> However, this patch reuses bind option 'ecdhe'. Currently it is
>> documented to accept only one curve. I believe it should be at least
>> updated to state that multiple curves can be used with sufficiently
>> new OpenSSL.
>> Also, I'm not sure what will happen when SSL_CTX_set1_curves_list() is
>> called with NULL (no ecdhe bind option). Even if it is accepted by
>> OpenSSL it will silently change haproxy default, before this patch it
>> was only prime256v1 (as defined in ECDHE_DEFAULT_CURVE), afterward it
>> will default to all curves supported by OpenSSL. Probably the best
>> would be to keep current default, so it all works consistently in
>> default configuration, regardless of version of haproxy and OpenSSL.
>>
>> --
>> Janusz Dziemidowicz
From a8b607bed1d787fdec67ad5234b678e4c1fbae72 Mon Sep 17 00:00:00 2001
From: David Martin 
Date: Thu, 14 Apr 2016 10:24:40 -0500
Subject: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

Use SSL_CTX_set_ecdh_auto if the OpenSSL version supports it, this
allows the server to negotiate ECDH curves much like it does ciphers.
Prefered curves can be specified using the existing ecdhe bind options
(ecdhe secp384r1:prime256v1)
---
 src/ssl_sock.c | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 0d35c29..a5d9408 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -2756,7 +2756,20 @@ int ssl_sock_prepare_ctx(struct bind_conf *bind_conf, SSL_CTX *ctx, struct proxy
 	SSL_CTX_set_tlsext_servername_callback(ctx, ssl_sock_switchctx_cbk);
 	SSL_CTX_set_tlsext_servername_arg(ctx, bind_conf);
 #endif
-#if defined(SSL_CTX_set_tmp_ecdh) && !defined(OPENSSL_NO_ECDH)
+#if !defined(OPENSSL_NO_ECDH)
+#if defined(SSL_CTX_set_ecdh_auto)
+	{
+		const char *ecdhe = (bind_conf->ecdhe ? bind_conf->ecdhe : ECDHE_DEFAULT_CURVE);
+		if (!SSL_CTX_set1_curves_list(ctx, ecdhe)) {
+			Alert("Proxy '%s': unable to set elliptic curve list to '%s' for bind '%s' at [%s:%d].\n",
+curproxy->id, ecdhe, bind_conf->arg, bind_conf->file, bind_conf->line);
+			cfgerr++;
+		}
+		else {
+			SSL_CTX_set_ecdh_auto(ctx, 1);
+		}
+	}
+#elif defined(SSL_CTX_set_tmp_ecdh)
 	{
 		int i;
 		EC_KEY  *ecdh;
@@ -2774,6 +2787,7 @@ int ssl_sock_prepare_ctx(struct bind_conf *bind_conf, SSL_CTX *ctx, struct proxy
 		}
 	}
 #endif
+#endif
 
 	return cfgerr;
 }
-- 
1.9.1

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[Janusz Dziemidowicz]

2016-04-14 12:05 GMT+02:00 Willy Tarreau :
> Hi David,
>
> On Wed, Apr 13, 2016 at 03:19:45PM -0500, David Martin wrote:
>> This is my first attempt at a patch, I'd love to get some feedback on this.
>>
>> Adds support for SSL_CTX_set_ecdh_auto which is available in OpenSSL 1.0.2.
>
>> From 05bee3e95e5969294998fb9e2794ef65ce5a6c1f Mon Sep 17 00:00:00 2001
>> From: David Martin 
>> Date: Wed, 13 Apr 2016 15:09:35 -0500
>> Subject: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection
>>
>> Use SSL_CTX_set_ecdh_auto if the OpenSSL version supports it, this
>> allows the server to negotiate ECDH curves much like it does ciphers.
>> Prefered curves can be specified using the existing ecdhe bind options
>> (ecdhe secp384r1:prime256v1)
>
> Could it have a performance impact ? I mean, may this allow a client to
> force the server to use curves that imply harder computations for example ?
> I'm asking because some people got seriously hit by the move from dhparm
> 1024 to 2048, so if this can come with a performance impact we possibly want
> to let the user configure it.

Switching ECDHE curves can have performance impact, for example result
of openssl speed on my laptop:
 256 bit ecdh (nistp256)   0.0003s   2935.3
 384 bit ecdh (nistp384)   0.0027s364.9
 521 bit ecdh (nistp521)   0.0016s623.2
The difference is so high for nistp256 because OpenSSL has heavily
optimized implementation
(https://www.imperialviolet.org/2010/12/04/ecc.html).

Apart from calling SSL_CTX_set_ecdh_auto() this patch also takes into
account user supplied curve list, so users can customize this as
needed (currently haproxy only allows to select one curve, which is a
limitation of older OpenSSL versions).

However, this patch reuses bind option 'ecdhe'. Currently it is
documented to accept only one curve. I believe it should be at least
updated to state that multiple curves can be used with sufficiently
new OpenSSL.
Also, I'm not sure what will happen when SSL_CTX_set1_curves_list() is
called with NULL (no ecdhe bind option). Even if it is accepted by
OpenSSL it will silently change haproxy default, before this patch it
was only prime256v1 (as defined in ECDHE_DEFAULT_CURVE), afterward it
will default to all curves supported by OpenSSL. Probably the best
would be to keep current default, so it all works consistently in
default configuration, regardless of version of haproxy and OpenSSL.

-- 
Janusz Dziemidowicz

Re: nbproc 1 vs >1 performance

[Willy Tarreau]

On Thu, Apr 14, 2016 at 01:54:27PM +0200, Daniel Schneller wrote:
> Trying not to hijack the thread here, but it seems to fit well in the context:
> 
> Does this mean that in the following could happen due to the difference in 
> BSD/Linux SO_REUSEPORT:
> 
> 1. haproxy process ???A??? binds say port 1234
> 2. client A connects to 1234 and keeps the connection open
> 3. /etc/init.d/haproxy restart
> 4. haproxy process ???B??? starts and _also_ binds 1234
> 5. haproxy ???A??? is still around, due to client A
> 6. client B connects to 1234
> 
> Am I right to assume that client B can be handled by _either_ haproxy ???A???
> or ???B??? depending on the hash result underneath SO_REUSEPORT???s
> implementation? If so, that would explain some issues I had in the past when
> quickly iterating config changes and restarting haproxy each time, but
> sometimes getting results that could only have come from an older config?

Not exactly. "haproxy restart" is "haproxy stop ; haproxy start" so it will
stop the old process. However "haproxy reload" will do that yes. But the
old process (process A) releases the port on reload, so new connections
will only go to the new process. There's a tiny fraction of second during
which both processes are bound to the same socket which ensures there's no
connection lost, but it lasts one millisecond or so.

Willy

Re: nbproc 1 vs >1 performance

[Daniel Schneller]

Trying not to hijack the thread here, but it seems to fit well in the context:

Does this mean that in the following could happen due to the difference in 
BSD/Linux SO_REUSEPORT:

1. haproxy process “A” binds say port 1234
2. client A connects to 1234 and keeps the connection open
3. /etc/init.d/haproxy restart
4. haproxy process “B” starts and _also_ binds 1234
5. haproxy “A” is still around, due to client A
6. client B connects to 1234

Am I right to assume that client B can be handled by _either_ haproxy “A” or 
“B” depending on the hash result underneath SO_REUSEPORT’s implementation? If 
so, that would explain some issues I had in the past when quickly iterating 
config changes and restarting haproxy each time, but sometimes getting results 
that could only have come from an older config?

Thanks,
Daniel


-- 
Daniel Schneller
Principal Cloud Engineer
CenterDevice GmbH


> On 14.04.2016, at 12:01, Willy Tarreau  wrote:
> 
> On Thu, Apr 14, 2016 at 10:17:10AM +0200, Willy Tarreau wrote:
>> So I guess that indeed, if not all the processes a frontend is bound to
>> have a corresponding bind line, this can cause connection issues as some
>> incoming connections will be distributed to queues that nobody listens to.
> 
> I said rubish here. It's the same socket that is shared between all these
> processes, so there's no issue with all of them picking from the same queue
> even if in the end only one picks it. However I want to fix it to make things
> cleaner and easier to debug and observe (and not fool netstat as the previous
> example showed).
> 
> Willy
> 

Re: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection

[Willy Tarreau]

Hi David,

On Wed, Apr 13, 2016 at 03:19:45PM -0500, David Martin wrote:
> This is my first attempt at a patch, I'd love to get some feedback on this.
> 
> Adds support for SSL_CTX_set_ecdh_auto which is available in OpenSSL 1.0.2.

> From 05bee3e95e5969294998fb9e2794ef65ce5a6c1f Mon Sep 17 00:00:00 2001
> From: David Martin 
> Date: Wed, 13 Apr 2016 15:09:35 -0500
> Subject: [PATCH] use SSL_CTX_set_ecdh_auto() for ecdh curve selection
> 
> Use SSL_CTX_set_ecdh_auto if the OpenSSL version supports it, this
> allows the server to negotiate ECDH curves much like it does ciphers.
> Prefered curves can be specified using the existing ecdhe bind options
> (ecdhe secp384r1:prime256v1)

Could it have a performance impact ? I mean, may this allow a client to
force the server to use curves that imply harder computations for example ?
I'm asking because some people got seriously hit by the move from dhparm
1024 to 2048, so if this can come with a performance impact we possibly want
to let the user configure it.

Thanks,
Willy

Re: nbproc 1 vs >1 performance

[Willy Tarreau]

On Thu, Apr 14, 2016 at 10:17:10AM +0200, Willy Tarreau wrote:
> So I guess that indeed, if not all the processes a frontend is bound to
> have a corresponding bind line, this can cause connection issues as some
> incoming connections will be distributed to queues that nobody listens to.

I said rubish here. It's the same socket that is shared between all these
processes, so there's no issue with all of them picking from the same queue
even if in the end only one picks it. However I want to fix it to make things
cleaner and easier to debug and observe (and not fool netstat as the previous
example showed).

Willy

Re: nbproc 1 vs >1 performance

[Willy Tarreau]

On Thu, Apr 14, 2016 at 11:49:47AM +0200, Christian Ruppert wrote:
> Yep, that did it. With this setting there is no more performance decrease on
> the http bind. Thanks!
> I'm just not sure if that will (negatively) affect anything else.

It may, depending on your setup (eg: if some frontends can drain many
connections at once it can increase latency).

Better apply the attached patch instead which divides by the number of
processes the listener is bound to :-)

Willy

>From 7c0ffd23d2d13e464a401292dcf55f658f2d3e24 Mon Sep 17 00:00:00 2001
From: Willy Tarreau 
Date: Thu, 14 Apr 2016 11:47:38 +0200
Subject: BUG/MEDIUM: fix maxaccept computation on per-process listeners

Christian Ruppert reported a performance degradation when binding a
single frontend to many processes while only one bind line was being
used, bound to a single process.

The reason comes from the fact that whenever a listener is bound to
multiple processes, the it is assigned a maxaccept value which equals
half the global maxaccept value divided by the number of processes the
frontend is bound to. The purpose is to ensure that no single process
will drain all the incoming requests at once and ensure a fair share
between all listeners. Usually this works pretty well, when a listener
is bound to all the processes of its frontend. But here we're in a
situation where the maxaccept of a listener which is bound to a single
process is still divided by a large value.

The fix consists in taking into account the number of processes the
listener is bound do and not only those of the frontend. This way it
is perfectly possible to benefit from nbproc and SO_REUSEPORT without
performance degradation.

1.6 and 1.5 normally suffer from the same issue.
---
 src/cfgparse.c | 14 ++
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/src/cfgparse.c b/src/cfgparse.c
index 2d0a020..c3b29d4 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -8694,9 +8694,6 @@ out_uri_auth_compat:
for (curproxy = proxy; curproxy; curproxy = curproxy->next) {
struct listener *listener;
unsigned int next_id;
-   int nbproc;
-
-   nbproc = my_popcountl(curproxy->bind_proc & 
nbits(global.nbproc));
 
 #ifdef USE_OPENSSL
/* Configure SSL for each bind line.
@@ -8741,6 +8738,15 @@ out_uri_auth_compat:
/* adjust this proxy's listeners */
next_id = 1;
list_for_each_entry(listener, >conf.listeners, by_fe) 
{
+   int nbproc;
+
+   nbproc = my_popcountl(curproxy->bind_proc &
+ listener->bind_conf->bind_proc &
+ nbits(global.nbproc));
+
+   if (!nbproc) /* no intersection between listener and 
frontend */
+   nbproc = 1;
+
if (!listener->luid) {
/* listener ID not set, use automatic numbering 
with first
 * spare entry starting with next_luid.
@@ -8819,7 +8825,7 @@ out_uri_auth_compat:
 #endif /* USE_OPENSSL */
}
 
-   if (nbproc > 1) {
+   if (my_popcountl(curproxy->bind_proc & nbits(global.nbproc)) > 
1) {
if (curproxy->uri_auth) {
int count, maxproc = 0;
 
-- 
1.7.12.1

Re: nbproc 1 vs >1 performance

[Willy Tarreau]

Hi Christian,

On Thu, Apr 14, 2016 at 11:06:02AM +0200, Christian Ruppert wrote:
> I've applied your patch and I just looked at the performance so far. The
> performance is still the same, so the lessperformant one is still less
> performant than the moreperformant.cfg. So from the performance point of
> view there's no difference between with and without that patch.
> 
> We have a (IMHO) quite huge haproxy config with around 200 frontends, ~180
> backends and ~90 listener. So the intention was to combine a http bind with
> a ssl bind in one frontend so that we keep it at a minimum that is necessary
> to get it working properly:
> listen ssl-relay
> mode tcp
> 
> bind-process 2
> 
> bind :443 process 2
> 
> tcp-request inspect-delay 7s
> acl HAS_ECC req.ssl_ec_ext eq 1
> tcp-request content accept if { req_ssl_hello_type 1 } # Client Hello
> 
> use-server ecc if HAS_ECC
> server ecc unix@/var/run/haproxy_ssl_ecc.sock send-proxy-v2
> 
> use-server rsa if !HAS_ECC
> server rsa unix@/var/run/haproxy_ssl_rsa.sock send-proxy-v2
> 
> 
> frontend http_https_combined
> mode http
> 
> bind-process 1-40
> 
> bind :80 process 1
> bind unix@/var/run/haproxy_ssl_ecc.sock accept-proxy ssl crt
> /etc/haproxy/ssltest.pem-ECC user haproxy process 4-40
> bind unix@/var/run/haproxy_ssl_rsa.sock accept-proxy ssl crt
> /etc/haproxy/ssltest.pem-RSA user haproxy process 3
> 
> ...
> 
> default_backend somebackend

OK I know why. It's because there's a per-listener "maxaccept" setting
which divides the global.tune.maxaccept value by 2 then by the number
of processes a listener is bound to. But in fact it divides by the number
of processes the frontend is bound to. In the past this used to be efficient
but since the "process" directive on frontends it doesn't make sense anymore.
So in practice above you end up with "maxaccept 1". You could try to work
around it by setting "tune.maxaccept 4000" in your global section, you'll
get the original performance back.

I have to change this obviously, so that we consider the intersection
between the listener and the frontend and not the frontend only. I'm
looking at this right now.

Willy

Re: nbproc 1 vs >1 performance

[Christian Ruppert]

Hi Willy,

On 2016-04-14 10:17, Willy Tarreau wrote:

On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:

Le me put it this way:

frontend haproxy_test
 bind-process 1-8
 bind :12345 process 1
 bind :12345 process 2
 bind :12345 process 3
 bind :12345 process 4


Leads to 8 processes, and the master process binds the socket 4 times 
(PID

16509):


(...)

lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
tcp0  0 *:12345 *:* LISTEN  
16509/haproxy
tcp0  0 *:12345 *:* LISTEN  
16509/haproxy
tcp0  0 *:12345 *:* LISTEN  
16509/haproxy
tcp0  0 *:12345 *:* LISTEN  
16509/haproxy

lukas@ubuntuvm:~/haproxy-1.5$


OK so it's netstat which gives a wrong report, I have the same here. I 
verified
in /proc/$PID/fd/ and I properly saw the FDs. Next, "ss -anp" also 
shows all the

process list :

  LISTEN 0  128   *:12345
  *:*
users:(("haproxy",25360,7),("haproxy",25359,7),("haproxy",25358,7),("haproxy",25357,7),("haproxy",25356,7),("haproxy",25355,7),("haproxy",25354,7),("haproxy",25353,7))
  LISTEN 0  128   *:12345
  *:*
users:(("haproxy",25360,6),("haproxy",25359,6),("haproxy",25358,6),("haproxy",25357,6),("haproxy",25356,6),("haproxy",25355,6),("haproxy",25354,6),("haproxy",25353,6))
  LISTEN 0  128   *:12345
  *:*
users:(("haproxy",25360,5),("haproxy",25359,5),("haproxy",25358,5),("haproxy",25357,5),("haproxy",25356,5),("haproxy",25355,5),("haproxy",25354,5),("haproxy",25353,5))
  LISTEN 0  128   *:12345
  *:*
users:(("haproxy",25360,4),("haproxy",25359,4),("haproxy",25358,4),("haproxy",25357,4),("haproxy",25356,4),("haproxy",25355,4),("haproxy",25354,4),("haproxy",25353,4))

A performance test also shows a fair distribution of the load :

  25353 willy 20   0 21872 4216 1668 S   26  0.1   0:04.54 haproxy
  25374 willy 20   0  7456  1080 S   25  0.0   0:02.26 
injectl464
  25376 willy 20   0  7456  1080 S   25  0.0   0:02.27 
injectl464
  25377 willy 20   0  7456  1080 S   25  0.0   0:02.26 
injectl464
  25375 willy 20   0  7456  1080 S   24  0.0   0:02.26 
injectl464

  25354 willy 20   0 21872 4168 1620 R   22  0.1   0:04.51 haproxy
  25356 willy 20   0 21872 4216 1668 R   22  0.1   0:04.21 haproxy
  25355 willy 20   0 21872 4168 1620 S   21  0.1   0:04.38 haproxy

However, as you can see these sockets are still bound to all processes 
and

that's not a good idea in the multi-queue mode.

I have added a few debug lines in enable_listener() like this :

$ git diff
diff --git a/src/listener.c b/src/listener.c
index 5abeb80..59c51a1 100644
--- a/src/listener.c
+++ b/src/listener.c
@@ -49,6 +49,7 @@ static struct bind_kw_list bind_keywords = {
  */
 void enable_listener(struct listener *listener)
 {
+   fddebug("%d: enabling fd %d\n", getpid(), listener->fd);
if (listener->state == LI_LISTEN) {
if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) &&
listener->bind_conf->bind_proc &&
@@ -57,6 +58,7 @@ void enable_listener(struct listener *listener)
 * want any fd event to reach it.
 */
fd_stop_recv(listener->fd);
+   fddebug("%d: pausing fd %d\n", getpid(), 
listener->fd);

listener->state = LI_PAUSED;
}
else if (listener->nbconn < listener->maxconn) {

And we're seeing this upon startup for processes 25746..25755 :

Thus as you can see that FDs are properly enabled and paused for the
unavailable ones.

willy@wtap:haproxy$ grep 4294967295 log | grep 25746
25746 write(4294967295, "25746: enabling fd 4\n", 21 
25746 write(4294967295, "25746: enabling fd 5\n", 21 
25746 write(4294967295, "25746: pausing fd 5\n", 20) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: enabling fd 6\n", 21) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: pausing fd 6\n", 20) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: enabling fd 7\n", 21 
25746 write(4294967295, "25746: pausing fd 7\n", 20 
willy@wtap:haproxy$ grep 4294967295 log | grep 25747
25747 write(4294967295, "25747: enabling fd 4\n", 21 
25747 write(4294967295, "25747: pausing fd 4\n", 20 
25747 write(4294967295, "25747: enabling fd 5\n", 21 
25747 write(4294967295, "25747: enabling fd 6\n", 21 
25747 write(4294967295, "25747: pausing fd 6\n", 20 
25747 write(4294967295, "25747: enabling fd 7\n", 21 
25747 write(4294967295, "25747: pausing fd 7\n", 20 
willy@wtap:haproxy$ grep 4294967295 log | grep 25748
25748 write(4294967295, "25748: enabling fd 4\n", 21 
25748 write(4294967295, "25748: pausing fd 4\n", 20 
25748 write(4294967295, "25748: enabling fd 5\n", 21 
25748 write(4294967295, "25748: pausing fd 5\n", 20 
25748 write(4294967295, 

Re: Haproxy & Kubernetes, dynamic backend configuration

[Smain Kahlouch]

Is it plan to support backend configuration with stats socket ?

2016-04-13 16:09 GMT+02:00 Smain Kahlouch :

> Ok thank you,
> I'll have a look to SmartStack.
>
> 2016-04-13 16:03 GMT+02:00 B. Heath Robinson :
>
>> SmartStack was mentioned earlier in the thread.  It does a VERY good job
>> of doing this.  It rewrites the haproxy configuration and performs a reload
>> based on changes in a source database by a polling service on each
>> instance.  The canonical DB is zookeeper.
>>
>> We have been using this in production for over 2 years and have had very
>> little trouble with it.  In the next year we will be moving to docker
>> containers and plan to make some changes to our configuration and move
>> forward with SmartStack.
>>
>> That said, your polling application could write instructions to the stats
>> socket; however, it currently does not allow adding/removing servers but
>> only enabling/disabling.  See
>> https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2 for
>> more info.  BTW, this section is missing from the 1.6 manual.  You might
>> also see https://github.com/flores/haproxyctl.
>>
>> On Wed, Apr 13, 2016 at 7:42 AM Smain Kahlouch 
>> wrote:
>>
>>> Sorry to answer to this thread so late :p
>>>
>>>
>>> due to the fact that this will be changed when the pod is recreated!

>>>
>>> Alexis, as i mentionned earlier the idea is to detect these changes by
>>> polling in a regular basis the API and change the backend configuration
>>> automatically.
>>> Using the DNS (addon) is not what i would like to achieve because it
>>> still uses kubernetes internal loadbalancing system.
>>> Furthermore it seems to me easier to use the NodePort,
>>> This is
>>> what i use today.
>>>
>>> Nginx Plus has now such feature :
>>>
 With APIs – This method uses the NGINX Plus on-the-fly reconfiguration
 API
 
 to add and remove entries for Kubernetes pods in the NGINX Plus
 configuration, and the Kubernetes API to retrieve the IP addresses of the
 pods. This method requires us to write some code, and we won’t discuss it
 in depth here. For details, watch Kelsey Hightower’s webinar, Bringing
 Kubernetes to the Edge with NGINX Plus
 ,
 in which he explores the APIs and creates an application that utilizes 
 them.

>>>
>>> Please let me know if you are considering this feature in the future.
>>> Alternatively perhaps you can guide me to propose a plugin. Actually
>>> python is the language i used to play with but maybe that's not possible.
>>>
>>> Regards,
>>> Smana
>>>
>>> 2016-02-25 18:29 GMT+01:00 Aleksandar Lazic :
>>>
 Hi.

 Am 25-02-2016 16:15, schrieb Smain Kahlouch:

> Hi !
>
> Sorry to bother you again with this question, but still i think it
> would
> be a great feature to loadbalance directly to pods from haproxy :)
> Is there any news on the roadmap about that ?
>

 How about DNS as mentioned below?


 https://github.com/kubernetes/kubernetes/blob/v1.0.6/cluster/addons/dns/README.md
 http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#5.3

 ### oc rsh -c ng-socklog nginx-test-2-6em5w
 cat /etc/resolv.conf
 nameserver 172.30.0.1
 nameserver 172.31.31.227
 search nginx-test.svc.cluster.local svc.cluster.local cluster.local
 options ndots:5

 ping docker-registry.default.svc.cluster.local
 

 
 oc describe svc docker-registry -n default
 Name:   docker-registry
 Namespace:  default
 Labels: docker-registry=default
 Selector:   docker-registry=default
 Type:   ClusterIP
 IP: 172.30.38.182
 Port:   5000-tcp5000/TCP
 Endpoints:  10.1.5.52:5000
 Session Affinity:   None
 No events.
 

 Another option is that you startup script adds the A record into skydns

 https://github.com/skynetservices/skydns

 But I don't see benefit to conncect directly to the endpoint, due to
 the fact that this will be changed when the pod is recreated!

 BR Aleks

 Regards,
> Smana
>
> 2015-09-22 20:21 GMT+02:00 Joseph Lynch :
>
> Disclaimer: I help maintain SmartStack and this is a shameless plug
>>
>> You can also achieve a fast and reliable dynamic backend system by
>> using something off the shelf like airbnb/Yelp SmartStack
>> (http://nerds.airbnb.com/smartstack-service-discovery-cloud/).
>>
>> Basically there is nerve that runs on every 

Re: nbproc 1 vs >1 performance

[Willy Tarreau]

On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:
> Le me put it this way:
>
> frontend haproxy_test
>  bind-process 1-8
>  bind :12345 process 1
>  bind :12345 process 2
>  bind :12345 process 3
>  bind :12345 process 4
>
>
> Leads to 8 processes, and the master process binds the socket 4 times (PID
> 16509):
>
(...)
> lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
> tcp0  0 *:12345 *:* LISTEN  16509/haproxy
> tcp0  0 *:12345 *:* LISTEN  16509/haproxy
> tcp0  0 *:12345 *:* LISTEN  16509/haproxy
> tcp0  0 *:12345 *:* LISTEN  16509/haproxy
> lukas@ubuntuvm:~/haproxy-1.5$

OK so it's netstat which gives a wrong report, I have the same here. I verified
in /proc/$PID/fd/ and I properly saw the FDs. Next, "ss -anp" also shows all the
process list :

  LISTEN 0  128   *:12345*:*
  
users:(("haproxy",25360,7),("haproxy",25359,7),("haproxy",25358,7),("haproxy",25357,7),("haproxy",25356,7),("haproxy",25355,7),("haproxy",25354,7),("haproxy",25353,7))
  LISTEN 0  128   *:12345*:*
  
users:(("haproxy",25360,6),("haproxy",25359,6),("haproxy",25358,6),("haproxy",25357,6),("haproxy",25356,6),("haproxy",25355,6),("haproxy",25354,6),("haproxy",25353,6))
  LISTEN 0  128   *:12345*:*
  
users:(("haproxy",25360,5),("haproxy",25359,5),("haproxy",25358,5),("haproxy",25357,5),("haproxy",25356,5),("haproxy",25355,5),("haproxy",25354,5),("haproxy",25353,5))
  LISTEN 0  128   *:12345*:*
  
users:(("haproxy",25360,4),("haproxy",25359,4),("haproxy",25358,4),("haproxy",25357,4),("haproxy",25356,4),("haproxy",25355,4),("haproxy",25354,4),("haproxy",25353,4))

A performance test also shows a fair distribution of the load :

  25353 willy 20   0 21872 4216 1668 S   26  0.1   0:04.54 haproxy
  25374 willy 20   0  7456  1080 S   25  0.0   0:02.26 injectl464
  25376 willy 20   0  7456  1080 S   25  0.0   0:02.27 injectl464
  25377 willy 20   0  7456  1080 S   25  0.0   0:02.26 injectl464
  25375 willy 20   0  7456  1080 S   24  0.0   0:02.26 injectl464
  25354 willy 20   0 21872 4168 1620 R   22  0.1   0:04.51 haproxy
  25356 willy 20   0 21872 4216 1668 R   22  0.1   0:04.21 haproxy
  25355 willy 20   0 21872 4168 1620 S   21  0.1   0:04.38 haproxy

However, as you can see these sockets are still bound to all processes and
that's not a good idea in the multi-queue mode.

I have added a few debug lines in enable_listener() like this :

$ git diff
diff --git a/src/listener.c b/src/listener.c
index 5abeb80..59c51a1 100644
--- a/src/listener.c
+++ b/src/listener.c
@@ -49,6 +49,7 @@ static struct bind_kw_list bind_keywords = {
  */
 void enable_listener(struct listener *listener)
 {
+   fddebug("%d: enabling fd %d\n", getpid(), listener->fd);
if (listener->state == LI_LISTEN) {
if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) &&
listener->bind_conf->bind_proc &&
@@ -57,6 +58,7 @@ void enable_listener(struct listener *listener)
 * want any fd event to reach it.
 */
fd_stop_recv(listener->fd);
+   fddebug("%d: pausing fd %d\n", getpid(), listener->fd);
listener->state = LI_PAUSED;
}
else if (listener->nbconn < listener->maxconn) {

And we're seeing this upon startup for processes 25746..25755 :

Thus as you can see that FDs are properly enabled and paused for the
unavailable ones.

willy@wtap:haproxy$ grep 4294967295 log | grep 25746
25746 write(4294967295, "25746: enabling fd 4\n", 21 
25746 write(4294967295, "25746: enabling fd 5\n", 21 
25746 write(4294967295, "25746: pausing fd 5\n", 20) = -1 EBADF (Bad file 
descriptor)
25746 write(4294967295, "25746: enabling fd 6\n", 21) = -1 EBADF (Bad file 
descriptor)
25746 write(4294967295, "25746: pausing fd 6\n", 20) = -1 EBADF (Bad file 
descriptor)
25746 write(4294967295, "25746: enabling fd 7\n", 21 
25746 write(4294967295, "25746: pausing fd 7\n", 20 
willy@wtap:haproxy$ grep 4294967295 log | grep 25747
25747 write(4294967295, "25747: enabling fd 4\n", 21 
25747 write(4294967295, "25747: pausing fd 4\n", 20 
25747 write(4294967295, "25747: enabling fd 5\n", 21 
25747 write(4294967295, "25747: enabling fd 6\n", 21 
25747 write(4294967295, "25747: pausing fd 6\n", 20 
25747 write(4294967295, "25747: enabling fd 7\n", 21 
25747 write(4294967295, "25747: pausing fd 7\n", 20 
willy@wtap:haproxy$ grep 4294967295 log | grep 25748
25748 write(4294967295, "25748: enabling fd 4\n", 21 
25748 write(4294967295, "25748: pausing fd 4\n", 20 
25748 write(4294967295, "25748: enabling fd 5\n", 21 
25748 

Re: nbproc 1 vs >1 performance

[Willy Tarreau]

On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:
> Hi Willy,
> 
> 
> Am 14.04.2016 um 07:08 schrieb Willy Tarreau:
> >Hi Lukas,
> >
> >On Thu, Apr 14, 2016 at 12:14:15AM +0200, Lukas Tribus wrote:
> >>For example, the following configuration load balances the traffic across
> >>all 40 processes, expected or not?
> >>
> >>frontend haproxy_test
> >>  bind-process 1-40
> >>  bind :12345 process 1
> >
> >It's not expected. What is indicated above is that the frontend will
> >exist for the first 40 processes, and that port 12345 will be bound
> >only in process 1. Processes 2..40 will thus have no listener at all.
> 
> Well, we do only bind once here, not 40 times. But the traffic is then
> handled by all 40 processes.

No here it's handled by only one due to "bind :12345 process 1".

If there was not this "process 1", then it would indeed be handled by
all 40 processes in a random fashion.

> Le me put it this way:
> 
> frontend haproxy_test
>  bind-process 1-8
>  bind :12345 process 1
>  bind :12345 process 2
>  bind :12345 process 3
>  bind :12345 process 4
> 
> 
> Leads to 8 processes, and the master process binds the socket 4 times (PID
> 16509):
> 
> lukas@ubuntuvm:~/haproxy-1.5$ ps auxww | grep haproxy
> haproxy  16509  0.0  0.0  18460   320 ?Ss   08:41   0:00 ./haproxy
> -f ../cert/ruppert-nbproc-stress.cfg -D
> haproxy  16510  0.0  0.0  18460   320 ?Ss   08:41   0:00 ./haproxy
> -f ../cert/ruppert-nbproc-stress.cfg -D
> haproxy  16511  0.0  0.0  18460   320 ?Ss   08:41   0:00 ./haproxy
> -f ../cert/ruppert-nbproc-stress.cfg -D
> haproxy  16512  0.0  0.0  18460   320 ?Ss   08:41   0:00 ./haproxy
> -f ../cert/ruppert-nbproc-stress.cfg -D
> haproxy  16513  0.0  0.0  18460   320 ?Ss   08:41   0:00 ./haproxy
> -f ../cert/ruppert-nbproc-stress.cfg -D
> haproxy  16514  0.0  0.0  18460   320 ?Ss   08:41   0:00 ./haproxy
> -f ../cert/ruppert-nbproc-stress.cfg -D
> haproxy  16515  0.0  0.0  18460   320 ?Ss   08:41   0:00 ./haproxy
> -f ../cert/ruppert-nbproc-stress.cfg -D
> haproxy  16516  0.0  0.0  18460   320 ?Ss   08:41   0:00 ./haproxy
> -f ../cert/ruppert-nbproc-stress.cfg -D
> lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
> tcp0  0 *:12345 *:* LISTEN  16509/haproxy
> tcp0  0 *:12345 *:* LISTEN  16509/haproxy
> tcp0  0 *:12345 *:* LISTEN  16509/haproxy
> tcp0  0 *:12345 *:* LISTEN  16509/haproxy
> lukas@ubuntuvm:~/haproxy-1.5$

Then this is where we have a problem. Only processes 1..4 should be bound.

> >But I'm rather surprized, maybe we recently broke something because
> >we have a lot of people successfully running such configurations,
> >which is why I'm a bit surprized.
> 
> I can reproduce this in v1.5.0 too, I think it was always like that.

I'm still very surprized because what this means is that each time we
managed to make a config work with such a setup it was by pure luck.
The most common use-case is with SSL and I'm aware of a significant
number of deployments running this way with the expected results.

I'll have to run some local tests to understand what's going on.

Thanks,
Willy

Re: nbproc 1 vs >1 performance

[Lukas Tribus]

Hi Willy,


Am 14.04.2016 um 07:08 schrieb Willy Tarreau:

Hi Lukas,

On Thu, Apr 14, 2016 at 12:14:15AM +0200, Lukas Tribus wrote:
  

For example, the following configuration load balances the traffic across
all 40 processes, expected or not?

frontend haproxy_test
  bind-process 1-40
  bind :12345 process 1


It's not expected. What is indicated above is that the frontend will
exist for the first 40 processes, and that port 12345 will be bound
only in process 1. Processes 2..40 will thus have no listener at all.


Well, we do only bind once here, not 40 times. But the traffic is then 
handled by all 40 processes.



Le me put it this way:

frontend haproxy_test
 bind-process 1-8
 bind :12345 process 1
 bind :12345 process 2
 bind :12345 process 3
 bind :12345 process 4


Leads to 8 processes, and the master process binds the socket 4 times 
(PID 16509):


lukas@ubuntuvm:~/haproxy-1.5$ ps auxww | grep haproxy
haproxy  16509  0.0  0.0  18460   320 ?Ss   08:41   0:00 
./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D
haproxy  16510  0.0  0.0  18460   320 ?Ss   08:41   0:00 
./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D
haproxy  16511  0.0  0.0  18460   320 ?Ss   08:41   0:00 
./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D
haproxy  16512  0.0  0.0  18460   320 ?Ss   08:41   0:00 
./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D
haproxy  16513  0.0  0.0  18460   320 ?Ss   08:41   0:00 
./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D
haproxy  16514  0.0  0.0  18460   320 ?Ss   08:41   0:00 
./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D
haproxy  16515  0.0  0.0  18460   320 ?Ss   08:41   0:00 
./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D
haproxy  16516  0.0  0.0  18460   320 ?Ss   08:41   0:00 
./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D

lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
tcp0  0 *:12345 *:* LISTEN  16509/haproxy
tcp0  0 *:12345 *:* LISTEN  16509/haproxy
tcp0  0 *:12345 *:* LISTEN  16509/haproxy
tcp0  0 *:12345 *:* LISTEN  16509/haproxy
lukas@ubuntuvm:~/haproxy-1.5$



SO_REUSEPORT is almost irrelevant here. In fact it *happens* to perform
some load balancing, but the first use was simply to permit new processes
to rebind before unbinding older ones during reloads.


I know, haproxy unconditionally enables SO_REUSEPORT since 2006, because 
of OpenBSD reload/restarts (OpenBSD does not load-balance). But I'm not 
aware we ever changed anything after Linux introduced SO_REUSEPORT with 
load-balancing support in 3.9, so the fact that we bind everything to 
the master process is in my opinion simply based on the fact that 
SO_REUSEPORT when introduced was just a fix for reload/restart problems.





But I'm rather surprized, maybe we recently broke something because
we have a lot of people successfully running such configurations,
which is why I'm a bit surprized.


I can reproduce this in v1.5.0 too, I think it was always like that.




cheers,

lukas