HAProxy Healthcheck issue using Virtual hostname

[Sen]

Hi

I have an app deployed in Pivotal Cloudfoundry (PCF) and to route traffic
to an app in PCF, we have to use application route name (virtual hostname).

We have PCF in two different datacenters and I need to load balance the
traffic to these DCs , but I'm having the challenge in checking the health
of the application.

here is the config - when i have app deployed only in one DC:

backend CustomerLookupService.searchCustomer
mode http
balance roundrobin
option httpchk GET /env HTTP/1.1\r\nHost:\
customerlookup.xxx.px-prd02.cf..com:443\r\n
http-check expect rstatus ^200
option httplog
timeout server 60s
default-server inter 10s fall 3 rise 2
server s_CustomerLookupService.searchCustomer2
customerlookup.xxx.px-prd02.cf..com:443 check resolvers dns ssl verify
none


Now I need to route the traffic to "
customerlookup.xxx.px-prd03.cf..com:443
 " , in addition to "
customerlookup.xxx.px-prd02.cf..com:443 ".

in that case, how do I check the health of prd02 and prd03?.

I tried following - but it's not working:

backend CustomerLookupService.searchCustomer
mode http
balance roundrobin

option forwardfor

http-send-name-header Host
option httpchk GET /env HTTP/1.1
http-check expect rstatus ^200
option httplog
timeout server 60s
default-server inter 10s fall 3 rise 2
server s_CustomerLookupService.searchCustomer2
customerlookup.xxx.px-prd02.cf..com:443 check resolvers dns ssl verify
none
server s_CustomerLookupService.searchCustomer3
customerlookup.xxx.px-prd03.cf..com:443
 check resolvers dns
ssl verify none

Looks like, Hostname is not getting passed in healthcheck calls.

Any ideas how to solve this problem?

Thanks in advance
Sen.

Re: Use SNI with healthchecks

[Willy Tarreau]

On Fri, Apr 27, 2018 at 06:39:07AM +0200, Willy Tarreau wrote:
> I think that a few operators like strcmp() and concat() should be
> implemented to cover the short-term needs.

I forgot that I finally implemented concat() after talking about it for
about a year :-)  It is a good starting point to see how to implement
strcmp() if you're interested.

Willy

Re: Use SNI with healthchecks

[Willy Tarreau]

Hi Tim,

On Fri, Apr 27, 2018 at 12:16:15AM +0200, Tim Düsterhus wrote:
> The solution I got from "Holger Just" was:
> 
> > http-request   set-header X-CHECKSNI %[req.hdr(host)]==%[ssl_fc_sni] if 
> > { ssl_fc_has_sni }
> > http-request   deny  if { ssl_fc_has_sni } ! { 
> > hdr(X-CHECKSNI) -m reg -i ^(?.+)==\1$ }
> > http-request   del-header X-CHECKSNI
> 
> which works well, but is not intuitive and requires PCRE.
> 
> I have contemplated to add a patch with a `ssl_sni_http_host_mismatch`
> fetch, but I wasn't quite sure where to place it (it requires both http
> and ssl and thus is misplaced in both) and what parameters to pass
> (meaning SMP_USE_HRQHV et al). And thus I gave up.

I'd proceed differently I think. Just like we do have arithmetic operators
support variables as arguments, we could have string operators like strcmp()
which would support a variable as well. Then you could simply put your SNI
into a variable, and compare the host field with this variable.

In practice I've long wanted to have a stack-based sample expression
evaluation but while many of us think it will eventually appear, I think
that a few operators like strcmp() and concat() should be implemented to
cover the short-term needs.

> Not having this validation opens up possible security issues (validation
> of client certificates is based on SNI, routing is possibly based on the
> Host header).

For this specific case I agree.

> Personally I require SNI for everything and select the
> backends based on SNI and not the Host header.

This is dangerous because it's not what the backend servers will do,
so you need to keep this in mind to ensure you never switch to the
wrong place. Also as Lukas explained, when some certs overlap, you'll
only have the SNI matching the first Host. Apparently in your case it's
not an issue since you exactly compare them. Normally the reliable way
of doing it would be to ensure that SNI is valid (eg when you have to
check a client cert) but still rely on Host for routing.

> If you could give me some pointers on where to put such a fetch and what
> parameters to set I'd appreciate it. Or tell me that such a fetch is
> stupid, because it mixes information from different layers.

It's not stupid, it's just that I'm pretty sure that immediately after
being committed, someone will want to adjust its behaviour (ignore case,
stop on colon, etc). For this strcmp() will allow you to normalize each
party and then to compare the results.

Regards,
Willy

Re: Use SNI with healthchecks

[Willy Tarreau]

Hi Lukas,

On Fri, Apr 27, 2018 at 01:56:42AM +0200, Lukas Tribus wrote:
> Hello Willy,
> 
> 
> On 25 April 2018 at 12:16, Willy Tarreau  wrote:
> >> I'm not even sure that differentiate "Host" header from SNI values is
> >> possible on softwares like Nginx or Apache.
> >
> > It should not, that would be a violation of HTTP over TLS.
> 
> I think I disagree.
> 
> 
> This is very possible and in fact happens everyday. When the
> certificate has SAN's for www.example.org and mail.example.org, a
> browser uses a single TLS connection to access both domains - SNI will
> be whatever HTTP request warranted the TLS session in the first place
> (as SNI is only send in the TLS handshake's client_hello).

You're absolutely right.

> This only breaks when people try to content switch HTTP transactions
> that should be based on the Host header and look at the SNI header
> instead (which I'd argue is a layering violation). SNI should imho
> only be used for certificate selection at the TLS server
> endpoint/layer and best not be carried to upper layers.

Totally agreed, and I remember that both of us have been warning certain
people against the risks of using SNI for L7 switching.

> This is not limited to H2, I believe browsers do it in HTTP/1.1 as
> well, but here's the H2 RFC documenting this behavior [1]. H2 also
> introduces a new error code for servers unable to handle this, which
> is 421 [2]. Yes, at the end of RFC6066 section 3 [3] there is a SHOULD
> NOT about this. But it doesn't specifically address the HTTP use-case
> and it's not a MUST NOT. It doesn't make sense for a browser to throw
> away an already validated TLS sessions for which the hostname matches
> a SAN from the validated certificate.

I remember following a conversation on this exact subject on the HTTP WG,
though I didn't catch all the details. But it's true that in general
browsers will try hard to reuse existing connections as long as possible.

> Let's not make the FTP mess again and pull lower layer transport (or
> transport encryption) variables up into the application layer. Let the
> TLS server endpoint pick the certificate based on SNI and then stop
> looking at it would be my general advice.

Please note that we were discussing about the other way around, how to
automatically set SNI based on the Host header field, which is what
browsers and proxies normally do. However, this discussion made me
think about a case where we do not want to do this : if a browser
connects to a proxy using TLS (to protect user authentication), the
advertised SNI should not be the one from the Host header field but
the one matching the proxy host name, which doesn't appear in the
request and cannot be deduced. This is a good example showing that
we can't have SNI always deduced from the request.

> When TCP proxying TLS traffic and content switching based on SNI, one
> needs to make sure that the certificates do not overlap because of
> this connection reuse (just to provide another perspective), see [4]
> and [5].

Agreed. In my opinion, SNI-based switching is only usable when you know
that the certificates do not overlap. But in practice that's often the
case initially, until someone renews the certs with multiple SANs for
some purpose and starts to break everything.

Cheers,
Willy

Re: Use SNI with healthchecks

[Lukas Tribus]

Hello Willy,


On 25 April 2018 at 12:16, Willy Tarreau  wrote:
>> I'm not even sure that differentiate "Host" header from SNI values is
>> possible on softwares like Nginx or Apache.
>
> It should not, that would be a violation of HTTP over TLS.

I think I disagree.


This is very possible and in fact happens everyday. When the
certificate has SAN's for www.example.org and mail.example.org, a
browser uses a single TLS connection to access both domains - SNI will
be whatever HTTP request warranted the TLS session in the first place
(as SNI is only send in the TLS handshake's client_hello).

This only breaks when people try to content switch HTTP transactions
that should be based on the Host header and look at the SNI header
instead (which I'd argue is a layering violation). SNI should imho
only be used for certificate selection at the TLS server
endpoint/layer and best not be carried to upper layers.



This is not limited to H2, I believe browsers do it in HTTP/1.1 as
well, but here's the H2 RFC documenting this behavior [1]. H2 also
introduces a new error code for servers unable to handle this, which
is 421 [2]. Yes, at the end of RFC6066 section 3 [3] there is a SHOULD
NOT about this. But it doesn't specifically address the HTTP use-case
and it's not a MUST NOT. It doesn't make sense for a browser to throw
away an already validated TLS sessions for which the hostname matches
a SAN from the validated certificate.


Let's not make the FTP mess again and pull lower layer transport (or
transport encryption) variables up into the application layer. Let the
TLS server endpoint pick the certificate based on SNI and then stop
looking at it would be my general advice.


When TCP proxying TLS traffic and content switching based on SNI, one
needs to make sure that the certificates do not overlap because of
this connection reuse (just to provide another perspective), see [4]
and [5].



cheers,
lukas


[1] https://tools.ietf.org/html/rfc7540#section-9.1.1
[2] https://tools.ietf.org/html/rfc7540#section-9.1.2
[3] https://tools.ietf.org/html/rfc6066#section-3
[4] 
https://discourse.haproxy.org/t/sni-routing-to-servers-only-ever-sent-to-first-server/1265/4
[5] https://discourse.haproxy.org/t/h2-problem-with-multiple-backends/1650

Re: 1.9dev LUA shows partial results from print_r(core.get_info()) after adding headers ?

[PiBa-NL]

Hi Thierry,

Op 26-4-2018 om 12:25 schreef thierry.fourn...@arpalert.org:

Your trace shows a corrupted tree. Maybe it is due to the freebsd
architecture and the corruption is no reproductible on Linux ? I do not have
freebsd for testing.

Regards,
Thierry


My 'best' reproduction scenario involves around 50 to 100 request or 
sometimes less requests total made from my Chrome browser. (over a vpn 
connection which adds a little latency.. a ping/reply to the haproxy 
server takes +-17ms , maybe that helps reproduce.. :/ )
I've changed the lua function 'print_r' to not write anything in its 
wr() function, to rule out the the freebsd console / ssh session to be 
causing the issue. It stays the same though.


Also been adding some printf statements to how the eb_tree is used, 
which i think might be interesting..

    printf("eb32sc_insert(%d)\n",new);
    printf("eb32sc_next(%d)  leaf_p: %d \n",eb32, eb32->node.leaf_p);
    printf("eb32sc_delete(%d)\n",eb32);

The pattern i see here is that usually a task is inserted, the next task 
is looked up, and the task gets deleted again.
Last round before the crash i see that the task is inserted, then 
deleted and then afterwards the next task is being retrieved for that 
same 'item'.. Which fails.. Perhaps because it was the last task to 
run.?. And there is nolonger a higher root to jump higher up the tree.?. 
I must admit i don't fully grasp how the tree is traversed exactly.. I 
seem to see that at least for the first task to be put into the tree 
there is some special handling.. On the other side, if it aint in the 
tree, is there technically still a 'next' item??


Below the last part of that logging, and also attached the complete log 
from the start.. Perhaps it gives a clue.?.


Regards,

PiBa-NL (Pieter)

eb32sc_insert(35826016)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35826016)  leaf_p: 8606272
eb32sc_delete(35826016)
task_wakeup  35826016  32
active_tasks_mask |= t->thread_mask  (35826016)
eb32sc_insert(35826016)
task_wakeup  35826656  8
active_tasks_mask |= t->thread_mask  (35826656)
eb32sc_insert(35826656)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35826656)  leaf_p: 35826656
eb32sc_delete(35826656)
0013:TestSite.srvrep[0006:]: HTTP/1.1 200 OK
0013:TestSite.srvhdr[0006:]: Refresh: 1
0013:TestSite.srvhdr[0006:]: Server: haproxy/webstats
0013:TestSite.srvhdr[0006:]: Content-Type: text/html
0013:TestSite.srvhdr[0006:]: Content-Length: 1778
eb32sc_delete(35826016)
0013:TestSite.srvcls[0006:]
eb32sc_next(35826016)  leaf_p: 0
Segmentation fault (core dumped)

root@freebsd11:~/.netbeans/remote/192.168.8.93/pb3-Windows-x86_64/P/Git/haproxy 
# ./haproxy -f /home/thierry/git/haproxy/bug29.conf -d -dM0x55
Note: setting global.maxconn to 2000.
Available polling systems :
   poll : pref=200,  test result OK
 select : pref=150,  test result FAILED
 kqueue : disabled,  test result OK
Total: 3 (1 usable), will use poll.

Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
Using poll() as the polling mechanism.
task_wakeup  35824896  4
active_tasks_mask |= t->thread_mask  (35824896)
eb32sc_insert(35824896)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35824896)  leaf_p: 8606272
eb32sc_delete(35824896)
task_wakeup  35824896  8
active_tasks_mask |= t->thread_mask  (35824896)
eb32sc_insert(35824896)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35824896)  leaf_p: 8606272
eb32sc_delete(35824896)
task_wakeup  35825216  4
active_tasks_mask |= t->thread_mask  (35825216)
eb32sc_insert(35825216)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35825216)  leaf_p: 8606272
eb32sc_delete(35825216)
task_wakeup  35825216  8
active_tasks_mask |= t->thread_mask  (35825216)
eb32sc_insert(35825216)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35825216)  leaf_p: 8606272
eb32sc_delete(35825216)
[WARNING] 116/014322 (38016) : Server myservers/localSRVc is DOWN, reason: 
Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 
active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in 
queue.
task_wakeup  35824896  4
active_tasks_mask |= t->thread_mask  (35824896)
eb32sc_insert(35824896)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35824896)  leaf_p: 8606272
eb32sc_delete(35824896)
task_wakeup  35824896  8
active_tasks_mask |= t->thread_mask  (35824896)
eb32sc_insert(35824896)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35824896)  leaf_p: 8606272
eb32sc_delete(35824896)
task_wakeup  35825216  4
active_tasks_mask |= t->thread_mask  (35825216)
eb32sc_insert(35825216)
process_runnable_tasks() active_tasks_mask &= ~tid_bit
eb32sc_next(35825216)  leaf_p: 8606272
eb32sc_delete(35825216)
task_wakeup  35825216  8
active_tasks_mask |= t->thread_mask  (35825216)

http healthcheck on backend with multiple servers not working

[Ajay Kumar]

I need to perform HTTP healthcheck on individual backend servers and load 
balance among active backend servers.This works fine with single backend server 
when we mention host name in httpchk (option httpchk GET  /info HTTP/1.1 Host:\ 
abc.mysrv1.com) 
However i am unable to successfully configure it with multiple backend servers 
as i can't give individual host name on httpchk command and with below backend 
configuration getting http 400 error - 

-backend svc.op    balance roundrobin    mode 
http    http-send-name-header Host    option httpchk GET  /info HTTP/1.1    
http-check send-state    http-request set-uri /test/abcService    server 
abc.mysrv1.com abc.mysrv1.com:80 check    server abc.mysrv2.com  
abc.mysrv2.com:80 check-

[WARNING] 115/162226 (4788) : Server svc.op/abc.mysrv1.com is DOWN, reason: 
Layer7 wrong status, code: 400, info: "Bad Request: missing required Host 
header", check duration: 1ms. 1 active and 0 backup servers left. 0 sessions 
active, 0 requeued, 0 remaining in queue.[WARNING] 115/162227 (4788) : Server 
svc.op/abc.mysrv2.com is DOWN, reason: Layer7 wrong status, code: 400, info: 
"Bad Request: missing required Host header", check duration: 0ms. 0 active and 
0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in 
queue.[ALERT] 115/162227 (4788) : backend ' svc.op' has no server available!

$ ./haproxy -version
HA-Proxy version 1.6.7 2016/07/13
For Reference - I see a patch was created 
https://www.mail-archive.com/haproxy@formilux.org/msg17321.html to send backend 
host and port in httpchk to perform health check on individual backend servers.

ThanksTapay

Re: Use SNI with healthchecks

[Tim Düsterhus]

Willy,

Am 25.04.2018 um 12:16 schrieb Willy Tarreau:
> On Wed, Apr 25, 2018 at 09:48:13AM +, GALLISSOT VINCENT wrote:
>> I don't see a case were one would define a different check-sni or sni values
>> from the "Host" header.
> 
> It definitely must match in HTTP. *snip*
> 
>> I'm not even sure that differentiate "Host" header from SNI values is
>> possible on softwares like Nginx or Apache.
> 
> It should not, that would be a violation of HTTP over TLS.
> 

Unfortunately it’s not exactly easy to detect this condition using
haproxy. I asked on May 7th, 2017 on the list (thread: Compare against
variable string in ACL, Message-ID:
,
https://www.mail-archive.com/haproxy@formilux.org/msg26014.html).

The solution I got from "Holger Just" was:

>   http-request   set-header X-CHECKSNI %[req.hdr(host)]==%[ssl_fc_sni] if 
> { ssl_fc_has_sni }
>   http-request   deny  if { ssl_fc_has_sni } ! { 
> hdr(X-CHECKSNI) -m reg -i ^(?.+)==\1$ }
>   http-request   del-header X-CHECKSNI

which works well, but is not intuitive and requires PCRE.

I have contemplated to add a patch with a `ssl_sni_http_host_mismatch`
fetch, but I wasn't quite sure where to place it (it requires both http
and ssl and thus is misplaced in both) and what parameters to pass
(meaning SMP_USE_HRQHV et al). And thus I gave up.

Not having this validation opens up possible security issues (validation
of client certificates is based on SNI, routing is possibly based on the
Host header). Personally I require SNI for everything and select the
backends based on SNI and not the Host header.

If you could give me some pointers on where to put such a fetch and what
parameters to set I'd appreciate it. Or tell me that such a fetch is
stupid, because it mixes information from different layers.

Best regards
Tim Düsterhus

Question on Caching.

[Andrew Smalley]

Hello Haproxy mailing list

I have been looking at caching technology and have found this

https://github.com/jiangwenyuan/nuster/

It claims to be a v1.7  / v1.8 branch fully compatible with haproxy
and indeed based on haproxy with the added capibility of having a
really fast cache as described here
https://github.com/jiangwenyuan/nuster/wiki/Web-cache-server-performance-benchmark:-nuster-vs-nginx-vs-varnish-vs-squid

It looks interesting but I would love some feedback please


Andruw Smalley

Loadbalancer.org Ltd.

www.loadbalancer.org
+1 888 867 9504 / +44 (0)330 380 1064
asmal...@loadbalancer.org

Leave a Review | Deployment Guides | Blog

Re: [PATCH] BUG/MINOR: lua/threads: Make lua's tasks sticky to the current thread

[Willy Tarreau]

On Thu, Apr 26, 2018 at 02:42:46PM +0200, Christopher Faulet wrote:
> Willy,
> 
> Here is a patch to fix a bug recently reported by Pieter in the lua part (in
> the thread ".1.9dev LUA core.tcp() cannot be used from different threads").

Applied, thanks!

Willy

http-response set-header is unreliable

[Tim Düsterhus]

Hi

I have got a frontend in mode http that sets various headers
unconditionally:

>   http-response  set-headerExpect-CT
> "max-age=3600; report-uri=\"https://xxx.report-uri.com/r/d/ct/reportOnly\";
>   http-response  set-headerExpect-Staple
> "max-age=3600; 
> report-uri=\"https://xxx.report-uri.com/r/d/staple/reportOnly\;; 
> includeSubDomains"
>   http-response  set-headerPublic-Key-Pins-Report-Only  
> "pin-sha256=\"Vjs8r4z+xxx+eWys=\"; pin-sha256=\"xxx/ltjyo=\"; 
> pin-sha256=\"xxx/uEtLMkBgFF2Fuihg=\"; 
> report-uri=\"https://xxx.report-uri.io/r/default/hpkp/reportOnly\;; 
> max-age=86400"
>   http-response  set-headerReferrer-Policy  "same-origin"
>   http-response  set-headerStrict-Transport-Security
> "max-age=31536000; includeSubDomains"
>   http-response  set-headerX-Content-Type-Options   nosniff
>   http-response  set-headerX-Frame-Options  SAMEORIGIN
>   http-response  set-headerX-XSS-Protection "1; 
> mode=block"

This frontend talks (among others) to a backend that also sets a header
unconditionally:

>   http-response set-header Content-Security-Policy "xxx report-uri 
> https://xxx.report-uri.com/r/d/csp/enforce;;

Sometimes haproxy does not set all the headers in a response (namely:
X-Frame-Options and X-XSS-Protection are sometimes missing):

> [timwolla@~]http -v https://example.com/ |grep 'X-'   17:24:24
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF41041A
> X-Content-Type-Options: nosniff
> X-Frame-Options: SAMEORIGIN
> X-XSS-Protection: 1; mode=block
> [timwolla@~]http -v https://example.com/ |grep 'X-'   17:24:49
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF46041D
> X-Content-Type-Options: nosniff
> [timwolla@~]http -v https://example.com/ |grep 'X-'   17:24:55
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF49041F
> X-Content-Type-Options: nosniff
> [timwolla@~]http -v https://example.com/ |grep 'X-'   17:24:57
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF4A0421
> X-Content-Type-Options: nosniff
> [timwolla@~]curl -I https://example.com/ |grep 'X-'   17:24:59
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
>   0 00 00 0  0  0 --:--:--  0:00:01 --:--:-- 0
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF4F0477
> X-Content-Type-Options: nosniff
> X-Frame-Options: SAMEORIGIN
> X-XSS-Protection: 1; mode=block
> [timwolla@~]curl -I https://example.com/ |grep 'X-'   17:25:05
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF530491
> X-Content-Type-Options: nosniff
> X-Frame-Options: SAMEORIGIN
> X-XSS-Protection: 1; mode=block
> [timwolla@~]curl -I https://example.com/ |grep 'X-'   17:25:07
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF5404B3
> X-Content-Type-Options: nosniff
> X-Frame-Options: SAMEORIGIN
> X-XSS-Protection: 1; mode=block
> [timwolla@~]http -v https://example.com/ |grep 'X-'   17:25:09
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF580598
> X-Content-Type-Options: nosniff
> X-Frame-Options: SAMEORIGIN
> X-XSS-Protection: 1; mode=block
> [timwolla@~]http -v https://example.com/ |grep 'X-'   17:25:12
> X-UA-Compatible: IE=edge
> X-Req-ID: EXAMPLE-5AE1EF66067F
> X-Content-Type-Options: nosniff

The logs for the first two requests:
> Apr 26 15:24:49 xxx haproxy[7565]: 2003:xxx:53728 [26/Apr/2018:15:24:49.681] 
> fe_https~ bk_xxx/nginx 0/0/1/252/253 200 16912 - -  11/8/0/1/0 0/0 
> {xxx|HTTPie/0.9.2} "GET / HTTP/1.1" EXAMPLE-5AE1EF41041A
> Apr 26 15:24:55 xxx haproxy[7565]: 2003:xxx:53730 [26/Apr/2018:15:24:55.034] 
> fe_https~ bk_xxx/nginx 0/0/0/203/203 200 16911 - -  10/7/0/1/0 0/0 
> {xxx|HTTPie/0.9.2} "GET / HTTP/1.1" EXAMPLE-5AE1EF46041D

The hex value in the request IDs is: %Ts%rt (thus there have only been
two requests in between those two).

I'm running haproxy 1.8.8 on Debian Stretch, installed from Debian
Backports. I've enabled http2. I don't run with threads:

> [root@~]haproxy -vv
> HA-Proxy version 1.8.8-1~bpo9+1 2018/04/19
> Copyright 2000-2018 Willy Tarreau 
> 
> Build options :
>   TARGET  = linux2628
>   CPU = generic
>   CC  = gcc
>   CFLAGS  = -g -O2 -fdebug-prefix-map=/build/haproxy-1.8.8=. 

Truly seamless reloads

[Veiko Kukk]

Hi,

According to 
https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ 
:


"The patchset has already been merged into the HAProxy 1.8 development 
branch and will soon be backported to HAProxy Enterprise Edition 1.7r1 
and possibly 1.6r2."


Has it been backported to 1.7 and/or 1.6?

If yes, then should seamless reload also work with multiprocess 
configurations? (nbproc > 1).


Thanks in advance,
Veiko

[PATCH] BUG/MINOR: lua/threads: Make lua's tasks sticky to the current thread

[Christopher Faulet]

Willy,

Here is a patch to fix a bug recently reported by Pieter in the lua part 
(in the thread ".1.9dev LUA core.tcp() cannot be used from different 
threads").


Thanks,
--
Christopher Faulet
>From b38e9bdf727073a6063f1c56f173a247969c6f9e Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Wed, 25 Apr 2018 10:34:45 +0200
Subject: [PATCH] BUG/MINOR: lua/threads: Make lua's tasks sticky to the
 current thread

PiBa-NL reported a bug with tasks registered in lua when HAProxy is started with
serveral threads. These tasks have not specific affinity with threads so they
can be woken up on any threads. So, it is impossbile for these tasks to handled
cosockets or applets, because cosockets and applets are sticky on the thread
which created them. It is forbbiden to manipulate a cosocket from another
thread.

So to fix the bug, tasks registered in lua are now sticky to the current
thread. Because these tasks can be registered before threads creation, the
affinity is set the first time a lua's task is processed.

This patch must be backported in HAProxy 1.8.
---
 src/hlua.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/hlua.c b/src/hlua.c
index da5611065..32199c964 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -5529,6 +5529,9 @@ static struct task *hlua_process_task(struct task *task)
 	struct hlua *hlua = task->context;
 	enum hlua_exec status;
 
+	if (task->thread_mask == MAX_THREADS_MASK)
+		task_set_affinity(task, tid_bit);
+
 	/* If it is the first call to the task, we must initialize the
 	 * execution timeouts.
 	 */
-- 
2.14.3

Re: 1.9dev LUA core.tcp() cannot be used from different threads

[Christopher Faulet]

Le 26/04/2018 à 12:01, Thierry Fournier a écrit :




On 26 Apr 2018, at 11:49, Christopher Faulet  wrote:

Le 25/04/2018 à 20:51, PiBa-NL a écrit :

Hi Christopher, Thierry,
Op 25-4-2018 om 11:30 schreef Christopher Faulet:

Oh, these tasks can be created before the threads creation... Ok, so
maybe the right way to fix the bug is to registered these tasks
without specific affinity and set it on the current thread the first
time the tasks are woken up.

Here is an updated (and untested) patch. Pieter, could you check it
please ?

Thierry, is there any way to create cosockets and applets from outside
a lua's task and then manipulate them in the task's context ?


Thanks, works for me.
I've only tested the last patch from Christopher, and that seems to do
the trick nicely for my situation.
If you guys come up with a different approach i'm happy to try that as
well instead.


Thanks Pieter for you feedback. Thierry, is this patch seems good for you ? Can 
we merge it ?



Yes. Thanks Christopher.


Ok, so I'll resend it outside this thread to let Willy merge it. Thanks.

--
Christopher Faulet

Re: [PATCH] MEDIUM: cli: Add multi-line mode support

[Willy Tarreau]

On Thu, Apr 26, 2018 at 01:48:27PM +0200, Aurélien Nephtali wrote:
> Willy,
> 
> On Thu, Apr 26, 2018 at 12:32 PM, Willy Tarreau  wrote:
> > Thanks for this. All of this looks OK to me. Please just let me know if
> > you want me to merge them now or if you expect other adjustments.
> 
> I think it's OK for me.

OK so it's merged now.

thanks!
Willy

[bug] http-reuse and TCP mode warning when using PROXY protocol

[Louis Chanouha]

Hello,

I set a global http-reuse safe.

HAProxy displays a warning for http-reuse and send-proxy combinaison on 
TCP mode backends, but http-reuse is active only on HTTP mode backends.


[WARNING] 115/135122 (26529) : config : proxy ' SMTPS_SUBMISSION' : 
connections to server 'f1' will have a PROXY protocol header announcing 
the first client's IP address while http-reuse is enabled and allows the 
same connection to be shared between multiple clients. It is strongly 
advised to disable 'send-proxy' and to use the 'forwardfor' option instead.


  defaults
                [...]
                http-reuse  safe

  listen SMTPS_SUBMISSION
    bind    0.0.0.0:587
    mode    tcp
    balance roundrobin
    option  tcplog
    server  f1 1.2.3.4 587 check send-proxy 
check-send-proxy


Sorry for my english

Louis

Re: [PATCH] MEDIUM: cli: Add multi-line mode support

[Aurélien Nephtali]

Willy,

On Thu, Apr 26, 2018 at 12:32 PM, Willy Tarreau  wrote:
> Thanks for this. All of this looks OK to me. Please just let me know if
> you want me to merge them now or if you expect other adjustments.

I think it's OK for me.

Thanks !

-- 
Aurélien Nephtali

Re: [PATCH] MEDIUM: cli: Add multi-line mode support

[Willy Tarreau]

Hi Aurélien,

On Thu, Apr 26, 2018 at 09:52:59AM +0200, Aurélien Nephtali wrote:
> Hello Willy,
> 
> Sorry for the delay.

no problem.

> Here are the three amended patches.
> 
> Changes:
> - update the documentation
> - add some comments regarding the detection of the payload pattern
>   and the input that is zero terminated
> - use appctx->chunk to gather data without using the trash
> - allow semicolons in a payload (NEW)
> - make "add map" payload parsing clearer
> - fix "set ssl ocsp-response" payload parsing
> - don't skip whitespaces at the beginning of the payload (the less it
>   is modified the better it is, I think)
> 
> "0001-wip.patch" is an incremental patch for the review.

Thanks for this. All of this looks OK to me. Please just let me know if
you want me to merge them now or if you expect other adjustments.

Willy

Re: 1.9dev LUA shows partial results from print_r(core.get_info()) after adding headers ?

[thierry . fournier]

On Wed, 25 Apr 2018 22:13:46 +0200
PiBa-NL  wrote:

> Hi Thierry,
> 
> Op 25-4-2018 om 11:19 schreef Thierry Fournier:
> > I extracted the part which dumps the ‘core.get_info()’, and I can’t 
> > reproduce
> > the segfault. I attach the extracted code. I use le lastest master branch.
> >
> I'm testing on master branch as well.
> I started over with the extracted bug29.conf and bug29.lua configuration 
> and the coredump still happens for me.
> 
> I do request all pages below more or less simultaneously with Chrome 
> browser, and then after a few seconds the crash happens..
>    http://haproxy:8000/webrequest/
>    http://haproxy:8000/webrequest/
>    http://haproxy:8000/webrequest/
>    http://haproxy:8000/haproxy?stats
> 
> I've also requested both urls with 'wrk' and then it does *not* crash 
> after several thousand requests to both.. There is something strange 
> there.. Though chrome does of course request the /favicon.ico sends way 
> more headers..
> 
> FYI im using :
> FreeBSD freebsd11 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 
> 21 02:08:28 UTC 2017 
> r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
> Also when using nokqueue the issue still happens.. (for some things that 
> helps.. not today.)
> 
> Attached the exact sequence of events when it happened fast. And the 
> coredump backtrace belonging with that output.
> Anything i can try to narrow it down further.? Or perhaps leave it for 
> now.?. as long as i dont output tons of info on console after the last 
> tcp.send it seams to work okay for now..


I can't reproduce it. I tried with many request on the Lua page (1000/s),
in the same time any request on the stat page (1600/s) and request throught
the proxy (1000/s - with an haproxy wich return 302).

HAProxy doesn't want to crash.

Your trace shows a corrupted tree. Maybe it is due to the freebsd
architecture and the corruption is no reproductible on Linux ? I do not have
freebsd for testing.

Regards,
Thierry

Re: Persisting stick tables on reload on 1.8

[Christian Greger]

Hi,

Thanks, it's working now. I saw peers mentioned elsewhere, but disregarded
that section since i was testing on a single instance.

On Thu, Apr 26, 2018 at 11:22 AM, Lukas Tribus  wrote:

> Hello Christian,
>
>
> On 26 April 2018 at 09:45, Christian Greger  wrote:
>
>> Hi,
>>
>> I was hoping the seamless reload in 1.8 would retain stick tables, but
>> I'm having no luck. Is it possible?
>>
>
>
> Stick tables can be transferred from the old to the new process while
> reloading by configuring the peers section with a local hostname, this is
> supported since haproxy 1.5:
>
> http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#3.5-peer
>
>
>
> 1.8's seamless reload feature is about the actual listening sockets only.
>
>
>
> cheers,
> lukas
>
>


-- 

Christian Greger
977 65 075/ luxi2...@gmail.com

[image: Twitter]  [image: Google Plus]
 [image: Skype]

This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended recipient(s).
Any unauthorized disclosure, dissemination, distribution, copying or the
taking of any action in reliance on the information herein is prohibited.
E-mails are not secure and cannot be guaranteed to be error free as they
can be intercepted, amended, or contain viruses. Anyone who communicates
with us by e-mail is deemed to have accepted these risks. Company Name is
not responsible for errors or omissions in this message and denies any
responsibility for any damage arising from the use of e-mail. Any opinion
and other statement contained in this message and any attachment are solely
those of the author and do not necessarily represent those of the company.

Re: 1.9dev LUA core.tcp() cannot be used from different threads

[Thierry Fournier]

> On 26 Apr 2018, at 11:49, Christopher Faulet  wrote:
> 
> Le 25/04/2018 à 20:51, PiBa-NL a écrit :
>> Hi Christopher, Thierry,
>> Op 25-4-2018 om 11:30 schreef Christopher Faulet:
>>> Oh, these tasks can be created before the threads creation... Ok, so
>>> maybe the right way to fix the bug is to registered these tasks
>>> without specific affinity and set it on the current thread the first
>>> time the tasks are woken up.
>>> 
>>> Here is an updated (and untested) patch. Pieter, could you check it
>>> please ?
>>> 
>>> Thierry, is there any way to create cosockets and applets from outside
>>> a lua's task and then manipulate them in the task's context ?
>>> 
>> Thanks, works for me.
>> I've only tested the last patch from Christopher, and that seems to do
>> the trick nicely for my situation.
>> If you guys come up with a different approach i'm happy to try that as
>> well instead.
> 
> Thanks Pieter for you feedback. Thierry, is this patch seems good for you ? 
> Can we merge it ?


Yes. Thanks Christopher.

Thierry

Re: 1.9dev LUA core.tcp() cannot be used from different threads

[Christopher Faulet]

Le 25/04/2018 à 20:51, PiBa-NL a écrit :

Hi Christopher, Thierry,

Op 25-4-2018 om 11:30 schreef Christopher Faulet:

Oh, these tasks can be created before the threads creation... Ok, so
maybe the right way to fix the bug is to registered these tasks
without specific affinity and set it on the current thread the first
time the tasks are woken up.

Here is an updated (and untested) patch. Pieter, could you check it
please ?

Thierry, is there any way to create cosockets and applets from outside
a lua's task and then manipulate them in the task's context ?


Thanks, works for me.
I've only tested the last patch from Christopher, and that seems to do
the trick nicely for my situation.
If you guys come up with a different approach i'm happy to try that as
well instead.



Thanks Pieter for you feedback. Thierry, is this patch seems good for 
you ? Can we merge it ?


--
Christopher Faulet

Re: Persisting stick tables on reload on 1.8

[Lukas Tribus]

Hello Christian,


On 26 April 2018 at 09:45, Christian Greger  wrote:

> Hi,
>
> I was hoping the seamless reload in 1.8 would retain stick tables, but I'm
> having no luck. Is it possible?
>


Stick tables can be transferred from the old to the new process while
reloading by configuring the peers section with a local hostname, this is
supported since haproxy 1.5:

http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#3.5-peer



1.8's seamless reload feature is about the actual listening sockets only.



cheers,
lukas

Re: multithreading issuse in haproxy 1.8.5

[Willy Tarreau]

On Thu, Apr 26, 2018 at 10:58:07AM +0300, Slawa Olhovchenkov wrote:
> > > I am mean in case of dedicated listen socket pooler also can be
> > > dedicated, for load planing. For example:
> > > 
> > > frontend tcp1
> > > bind x.x.x.206:443
> > > bind-process 1/9-1/16
> > > mode tcp
> > > 
> > > threads 1-8 don't need any events from this socket at all.
> > 
> > That's exactly what happens.
> 
> No, as I see. After start load on this sockets I am see CPU use on CPU's
> 0-7 too. All CPU rise load simultaneous.

It should not be the case, but now that I'm thinking about it, it would
in fact be the same boot race as the issue on the DNS : the FDs are
registered before forking the threads, and then we have to update the
relevant pollers in each thread. But the information used to know what
state the FD is in to know if we need to perform an update is shared,
so depending on the boot order, the state can be updated on one poller
preventing the other ones from getting it. That's what we're working on.
Now I fear that we'll have to perform deeper changes to 1.8 than what we
initially expected :-/

> May be I am miss some in config?

No, your config is simple and OK.

Willy

Re: multithreading issuse in haproxy 1.8.5

[Slawa Olhovchenkov]

On Thu, Apr 26, 2018 at 09:49:53AM +0200, Willy Tarreau wrote:

> On Thu, Apr 26, 2018 at 10:35:51AM +0300, Slawa Olhovchenkov wrote:
> > On Thu, Apr 26, 2018 at 09:25:59AM +0200, Willy Tarreau wrote:
> > 
> > > On Thu, Apr 26, 2018 at 10:21:27AM +0300, Slawa Olhovchenkov wrote:
> > > > > > Pollers distinct from frontend?
> > > > > > Can I bind pollers to CPU?
> > > > > 
> > > > > Each thread has its own poller. Since you map threads to CPUs you 
> > > > > indeed
> > > > > have one poller per CPU.
> > > > 
> > > > Each pooler pool all sockets or only sockets from binded frontends?
> > > 
> > > All sockets. All FDs in fact. This is normal, it's an event loop, it needs
> > > to be notified of *any* event (fd activity, signal).
> > 
> > I am mean in case of dedicated listen socket pooler also can be
> > dedicated, for load planing. For example:
> > 
> > frontend tcp1
> > bind x.x.x.206:443
> > bind-process 1/9-1/16
> > mode tcp
> > 
> > threads 1-8 don't need any events from this socket at all.
> 
> That's exactly what happens.

No, as I see. After start load on this sockets I am see CPU use on CPU's
0-7 too. All CPU rise load simultaneous.

May be I am miss some in config?

> > This is also reduce communication w/ kernel, rise locality of data.
> > 
> > I mean locality accpeted socket only to one pooler will be good too.
> 
> It's what is done, don't worry. Please take a look at the code, namely
> fdtab[] in fd.h. You'll see a polled_mask and thread_mask for each fd,
> used to know what thread needs knowledge of the fd and what thread's
> poller currently polls the fd.
> 
> Willy

Re: [PATCH] MEDIUM: cli: Add multi-line mode support

[Aurélien Nephtali]

Hello Willy,

Sorry for the delay.

Here are the three amended patches.

Changes:
- update the documentation
- add some comments regarding the detection of the payload pattern
  and the input that is zero terminated
- use appctx->chunk to gather data without using the trash
- allow semicolons in a payload (NEW)
- make "add map" payload parsing clearer
- fix "set ssl ocsp-response" payload parsing
- don't skip whitespaces at the beginning of the payload (the less it
  is modified the better it is, I think)

"0001-wip.patch" is an incremental patch for the review.

Thanks.

-- 
Aurélien.
From 1c7b61400e1647cb0e9dd783375f24336352f464 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Aur=C3=A9lien=20Nephtali?= 
Date: Wed, 18 Apr 2018 13:26:46 +0200
Subject: [PATCH 1/3] MEDIUM: cli: Add payload support
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

In order to use arbitrary data in the CLI (multiple lines or group of words
that must be considered as a whole, for example), it is now possible to add a
payload to the commands. To do so, the first line needs to end with a special
pattern: <<\n. Everything that follows will be left untouched by the CLI parser
and will be passed to the commands parsers.

Per-command support will need to be added to take advantage of this
feature.

Signed-off-by: Aurélien Nephtali 
---
 doc/management.txt |  12 +++
 include/proto/applet.h |   4 +-
 include/proto/cli.h|   1 -
 include/types/applet.h |   6 +-
 include/types/cli.h|   2 +-
 src/cache.c|   2 +-
 src/cli.c  | 281 +++--
 src/dns.c  |   2 +-
 src/hlua.c |   2 +-
 src/map.c  |  12 +--
 src/proto_http.c   |   2 +-
 src/proxy.c|  16 +--
 src/server.c   |  20 ++--
 src/ssl_sock.c |   6 +-
 src/stats.c|   6 +-
 src/stick_table.c  |   2 +-
 src/stream.c   |   6 +-
 17 files changed, 237 insertions(+), 145 deletions(-)

diff --git a/doc/management.txt b/doc/management.txt
index 4b6901851..bca0fd469 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -1298,6 +1298,18 @@ delimiter to mark an end of output for each command, and takes care of ensuring
 that no command can emit an empty line on output. A script can thus easily
 parse the output even when multiple commands were pipelined on a single line.
 
+Some commands may take an optional payload. To add one to a command, the first
+line needs to end with the "<<\n" pattern. The next lines will be treated as
+the payload and can contain as many lines as needed. To validate a command with
+a payload, it needs to end with an empty line.
+
+Limitations do exist: the length of the whole buffer passed to the CLI must
+not be greater than tune.bfsize and the pattern "<<" must not be glued to the
+last word of the line.
+
+When entering a paylod while in interactive mode, the prompt will change from
+"> " to "+ ".
+
 It is important to understand that when multiple haproxy processes are started
 on the same sockets, any process may pick up the request and will output its
 own stats.
diff --git a/include/proto/applet.h b/include/proto/applet.h
index cdd4d90f0..7cc2c0ad1 100644
--- a/include/proto/applet.h
+++ b/include/proto/applet.h
@@ -43,11 +43,13 @@ static int inline appctx_res_wakeup(struct appctx *appctx);
 
 /* Initializes all required fields for a new appctx. Note that it does the
  * minimum acceptable initialization for an appctx. This means only the
- * 3 integer states st0, st1, st2 are zeroed.
+ * 3 integer states st0, st1, st2 and the chunk used to gather unfinished
+ * commands are zeroed
  */
 static inline void appctx_init(struct appctx *appctx, unsigned long thread_mask)
 {
 	appctx->st0 = appctx->st1 = appctx->st2 = 0;
+	appctx->chunk = NULL;
 	appctx->io_release = NULL;
 	appctx->thread_mask = thread_mask;
 	appctx->state = APPLET_SLEEPING;
diff --git a/include/proto/cli.h b/include/proto/cli.h
index d5feb862a..da80af7d3 100644
--- a/include/proto/cli.h
+++ b/include/proto/cli.h
@@ -24,7 +24,6 @@
 #define _PROTO_CLI_H
 
 
-struct cli_kw* cli_find_kw(char **args);
 void cli_register_kw(struct cli_kw_list *kw_list);
 
 int cli_has_level(struct appctx *appctx, int level);
diff --git a/include/types/applet.h b/include/types/applet.h
index 89c318c1d..b0715866e 100644
--- a/include/types/applet.h
+++ b/include/types/applet.h
@@ -50,6 +50,9 @@ struct applet {
 #define APPLET_WOKEN_UP 0x02  /* applet was running and requested to woken up again */
 #define APPLET_WANT_DIE 0x04  /* applet was running and requested to die */
 
+#define APPCTX_CLI_ST1_PROMPT  (1 << 0)
+#define APPCTX_CLI_ST1_PAYLOAD (1 << 1)
+
 /* Context of a running applet. */
 struct appctx {
 	struct list runq;  /* chaining in the applet run queue */
@@ -57,7 +60,8 @@ struct appctx {
 	/* 

Re: multithreading issuse in haproxy 1.8.5

[Willy Tarreau]

On Thu, Apr 26, 2018 at 10:35:51AM +0300, Slawa Olhovchenkov wrote:
> On Thu, Apr 26, 2018 at 09:25:59AM +0200, Willy Tarreau wrote:
> 
> > On Thu, Apr 26, 2018 at 10:21:27AM +0300, Slawa Olhovchenkov wrote:
> > > > > Pollers distinct from frontend?
> > > > > Can I bind pollers to CPU?
> > > > 
> > > > Each thread has its own poller. Since you map threads to CPUs you indeed
> > > > have one poller per CPU.
> > > 
> > > Each pooler pool all sockets or only sockets from binded frontends?
> > 
> > All sockets. All FDs in fact. This is normal, it's an event loop, it needs
> > to be notified of *any* event (fd activity, signal).
> 
> I am mean in case of dedicated listen socket pooler also can be
> dedicated, for load planing. For example:
> 
> frontend tcp1
> bind x.x.x.206:443
> bind-process 1/9-1/16
> mode tcp
> 
> threads 1-8 don't need any events from this socket at all.

That's exactly what happens.

> This is also reduce communication w/ kernel, rise locality of data.
> 
> I mean locality accpeted socket only to one pooler will be good too.

It's what is done, don't worry. Please take a look at the code, namely
fdtab[] in fd.h. You'll see a polled_mask and thread_mask for each fd,
used to know what thread needs knowledge of the fd and what thread's
poller currently polls the fd.

Willy

Persisting stick tables on reload on 1.8

[Christian Greger]

Hi,

I was hoping the seamless reload in 1.8 would retain stick tables, but I'm
having no luck. Is it possible?

I'm testing this with:

haproxy: 1.8.8, single master, single thread, nbproc 1
OS: CentOS 7.4.1708
socket: stats socket /var/run/haproxy.sock level admin expose-fd listeners

Systemctl status:
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor
preset: disabled)
   Active: active (running) since Wed 2018-04-25 19:28:21 CEST; 14h ago
  Process: 3504 ExecReload=/bin/kill -USR2 $MAINPID (code=exited,
status=0/SUCCESS)
  Process: 3503 ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited,
status=0/SUCCESS)
  Process: 3318 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q
(code=exited, status=0/SUCCESS)
 Main PID: 3319 (haproxy)
   CGroup:
/docker/b456dcda6def2ef537ee15d9083799094193d2a314102101a3dfbca66b72de46/system.slice/haproxy.service
   ├─3319 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -sf 3497 -x /var/run/haproxy.sock
   └─3505 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -sf 3497 -x /var/run/haproxy.sock

Apr 25 19:53:29 lb01 haproxy[3319]: [WARNING] 114/195328 (3319) : Former
worker 3320 exited with code 0
Apr 25 19:53:56 lb01 haproxy[3319]: [WARNING] 114/195328 (3319) :
Reexecuting Master process
Apr 25 19:53:56 lb01 systemd[1]: Reloaded HAProxy Load Balancer.
Apr 25 19:53:56 lb01 haproxy[3319]: [WARNING] 114/195356 (3319) : Former
worker 3406 exited with code 0
Apr 25 19:57:24 lb01 haproxy[3319]: [WARNING] 114/195356 (3319) :
Reexecuting Master process
Apr 25 19:57:24 lb01 systemd[1]: Reloaded HAProxy Load Balancer.

haproxy -vv:

HA-Proxy version 1.8.8 2018/04/19
Copyright 2000-2018 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
-Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
  OPTIONS = USE_SLZ=1 USE_REGPARM=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace


-- 

Christian Greger
977 65 075/ luxi2...@gmail.com

[image: Twitter]  [image: Google Plus]
 [image: Skype]

This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended recipient(s).
Any unauthorized disclosure, dissemination, distribution, copying or the
taking of any action in reliance on the information herein is prohibited.
E-mails are not secure and cannot be guaranteed to be error free as they
can be intercepted, amended, or contain viruses. Anyone who communicates
with us by e-mail is deemed to have accepted these risks. Company Name is
not responsible for errors or omissions in this message and denies any
responsibility for any damage arising from the use of e-mail. Any opinion
and other statement contained in this message and any attachment are solely
those of the author and do not necessarily represent those of the company.

Re: multithreading issuse in haproxy 1.8.5

[Slawa Olhovchenkov]

On Thu, Apr 26, 2018 at 09:25:59AM +0200, Willy Tarreau wrote:

> On Thu, Apr 26, 2018 at 10:21:27AM +0300, Slawa Olhovchenkov wrote:
> > > > Pollers distinct from frontend?
> > > > Can I bind pollers to CPU?
> > > 
> > > Each thread has its own poller. Since you map threads to CPUs you indeed
> > > have one poller per CPU.
> > 
> > Each pooler pool all sockets or only sockets from binded frontends?
> 
> All sockets. All FDs in fact. This is normal, it's an event loop, it needs
> to be notified of *any* event (fd activity, signal).

I am mean in case of dedicated listen socket pooler also can be
dedicated, for load planing. For example:

frontend tcp1
bind x.x.x.206:443
bind-process 1/9-1/16
mode tcp

threads 1-8 don't need any events from this socket at all.
This is also reduce communication w/ kernel, rise locality of data.

I mean locality accpeted socket only to one pooler will be good too.

> > > Please try this patch. It works for me. I finally managed to reproduce
> > > the issue even with epoll(), it's just that it's much harder to see it,
> > > but after trying multiple times eventually you see it as well. Under
> > > poll() however the issue occasionally happens and disappears by itself.
> > 
> > Like work for me too, thank
> 
> Great, thank you. You can use it for now to fix your production, it will
> likely be the one we'll use in 1.8 in the end. We're still working on
> addressing the root cause in 1.9 first.

10x

Re: multithreading issuse in haproxy 1.8.5

[Willy Tarreau]

On Thu, Apr 26, 2018 at 10:21:27AM +0300, Slawa Olhovchenkov wrote:
> > > Pollers distinct from frontend?
> > > Can I bind pollers to CPU?
> > 
> > Each thread has its own poller. Since you map threads to CPUs you indeed
> > have one poller per CPU.
> 
> Each pooler pool all sockets or only sockets from binded frontends?

All sockets. All FDs in fact. This is normal, it's an event loop, it needs
to be notified of *any* event (fd activity, signal).

> > Please try this patch. It works for me. I finally managed to reproduce
> > the issue even with epoll(), it's just that it's much harder to see it,
> > but after trying multiple times eventually you see it as well. Under
> > poll() however the issue occasionally happens and disappears by itself.
> 
> Like work for me too, thank

Great, thank you. You can use it for now to fix your production, it will
likely be the one we'll use in 1.8 in the end. We're still working on
addressing the root cause in 1.9 first.

Willy

Re: multithreading issuse in haproxy 1.8.5

[Slawa Olhovchenkov]

On Wed, Apr 25, 2018 at 03:49:09PM +0200, Willy Tarreau wrote:

> On Wed, Apr 25, 2018 at 04:24:42PM +0300, Slawa Olhovchenkov wrote:
> > > > TCP load rise CPU use on all core (0-15), I am expect rise CPU use
> > > > only on 8-15 core. What I am miss?
> > > 
> > > It's unrelated to the frontend's bindings but to the way the socket's fd
> > > is registered with pollers, which is why you still have the problem here.
> > > We're still surprized it doesn't happen with other pollers, which is why
> > > we need to dig deeper as it could cover another issue.
> > 
> > Pollers distinct from frontend?
> > Can I bind pollers to CPU?
> 
> Each thread has its own poller. Since you map threads to CPUs you indeed
> have one poller per CPU.

Each pooler pool all sockets or only sockets from binded frontends?

> > > We'll keep you updated.
> > 
> > Thanks so much
> 
> Please try this patch. It works for me. I finally managed to reproduce
> the issue even with epoll(), it's just that it's much harder to see it,
> but after trying multiple times eventually you see it as well. Under
> poll() however the issue occasionally happens and disappears by itself.

Like work for me too, thank

> Olivier found that we have a race condition in the way we update FDs that
> are polled by multiple pollers. This is a side effect of the per-thread
> poller change that we had to do recently to fix other issues. The good
> news is that in 1.8, only listeners should be present on multiple threads
> so it is not dramatic. The DNS has no reason for being present anywhere
> but on the thread that creates the outgoing connection. It's what this
> patches does. However now we have a painful job of trying to address
> the listener case, as I think there are definitely races there that are
> quite hard to reach but we don't want to leave them.
> 
> Willy
> 

> diff --git a/src/dns.c b/src/dns.c
> index b1fac10..d3b2c10 100644
> --- a/src/dns.c
> +++ b/src/dns.c
> @@ -195,7 +195,7 @@ static int dns_connect_namesaver(struct dns_nameserver 
> *ns)
>   dgram->t.sock.fd = fd;
>   fdtab[fd].owner  = dgram;
>   fdtab[fd].iocb   = dgram_fd_handler;
> - fd_insert(fd, (unsigned long)-1);
> + fd_insert(fd, tid_bit);
>   fd_want_recv(fd);
>   return 0;
>  }