Lock contention in pat_match_str in threaded mode

[Brian Diekelman]

Just wanted to provide some information on what appears to be lock contention 
around ACL lookups.

We recently upgraded from haproxy-1.6 to haproxy-1.8.20 and switched from 
'nbprocs 8' to 'nbprocs 1, nbthreads 12'

We have quite a few ACLs files to sift through for domain matching -- about 
19MB total.

After switching to 1.8.20 we saw elevated CPU under our normal peak load, to 
the point where in some cases all 12 threads were consuming 100% CPU.

Looking at the 'perf' numbers:

  90.71%  haproxy  [.] pat_match_str
   0.49%  libc-2.12.so [.] __strncasecmp_l_sse42
   0.41%  haproxy  [.] pat_match_beg
   0.34%  haproxy  [.] lru64_get
   0.26%  [kernel] [k] _spin_lock_bh
   0.17%  haproxy  [.] pat_match_sub
   0.16%  libc-2.12.so [.] _int_malloc
   0.13%  [kernel] [k] _spin_lock
   0.10%  haproxy  [.] process_runnable_tasks
   0.09%  libc-2.12.so [.] malloc


We looked up that function, and it seemed like all requests were going through 
one spin lock [0].
The LRU tree seemed to be conditionally initialized based on 
'global.tune.pattern_cache' which seemed to be tied to the 
'tune.pattern.cache-size' config value [1] [2].
So, we specified 'tune.pattern.cache-size 0' to disable the cache code path and 
avoid the spinlock, and the CPU numbers dropped back down to where they had 
been previously (and lru64_get was no longer in the execution traces).

I realize our use case is probably an edge case (that's a lot of ACL data -- we 
have plans to upgrade these to map files now that we're on haproxy-1.8).
Just wanted to make the developers aware and help anyone who may run into this.

Thank you,
-Brian

[0] 
https://github.com/haproxy/haproxy/blob/33ccf1cce04069f27ebb868f5617672ed4e21cf4/src/pattern.c#L487
[1] 
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-tune.pattern.cache-size
[2] 
https://github.com/haproxy/haproxy/blob/33ccf1cce04069f27ebb868f5617672ed4e21cf4/src/pattern.c#L2688

Re: freebsd builds are broken for few days - 30ee1ef, proxy_protocol_random_fail.vtc fails because scheme and host are now present in the syslog output.

[PiBa-NL]

Hi Christopher,

It seems you fixed/changed the issue i noticed below a few minutes ago 
in commit 452e578 :) , thanks.
One question remaining about this on my side is if it is expected that 
some platforms will use 'normalized' URI and others platforms just the 
regular / ?


Regards, PiBa-NL (Pieter)

Op 14-10-2019 om 21:22 schreef PiBa-NL:

Hi Ilya, Willy,

Op 13-10-2019 om 19:30 schreef Илья Шипицин:

https://cirrus-ci.com/github/haproxy/haproxy

I'll bisect if noone else knows what's going on


@IIlya, thanks for checking my favorite platform, FreeBSD ;).

@Willy, this 30ee1ef 
 (MEDIUM: 
h2: use the normalized URI encoding for absolute form requests) commit 
'broke' the expect value of the vtest, i don't know why other 
platforms don't see the same change in syslog output though.. Anyhow 
this is the output i get when running the 
/reg-tests/connection/proxy_protocol_random_fail.vtc


 Slog_1  0.033 syslog|<134>Oct 14 19:56:34 haproxy[78982]: 
::1:47040 [14/Oct/2019:19:56:34.391] ssl-offload-http~ 
ssl-offload-http/http 0/0/0/0/0 503 222 - -  1/1/0/0/0 0/0 "POST 
https://[::1]:47037/1 HTTP/2.0"
**   Slog_1  0.033 === expect ~ "ssl-offload-http/http .* \"POST 
/[1-8] HTTP/(2\\.0...
 Slog_1  0.033 EXPECT FAILED ~ "ssl-offload-http/http .* "POST 
/[1-8] HTTP/(2\.0|1\.1)""


If i change the vtc vtest file from:
    expect ~ "ssl-offload-http/http .* \"POST /[1-8] 
HTTP/(2\\.0|1\\.1)\""

To:
    expect ~ "ssl-offload-http/http .* \"POST 
https://[[]::1]:[0-9]{1,5}/[1-8] HTTP/(2\\.0|1\\.1)\""

or:
    expect ~ "ssl-offload-http/http .* \"POST 
https://[[]${h1_ssl_addr}]:${h1_ssl_port}/[1-8] HTTP/(2\\.0|1\\.1)\""


Then the test succeeds for me... but now the question is, should or 
shouldn't the scheme and host be present in the syslog output on all 
platforms.? Or should the regex contain a (optional?) check for this 
extra part? (Also note that even with these added variables in my 
second regext attempt its still using accolades around the IPv6 
address.. not sure if all machines would use ipv6 for their localhost 
connection..)


Regards,
PiBa-NL (Pieter)

commit 246c024 - breaks loading crt-list with .ocsp files present

[PiBa-NL]

Hi William,

I'm having an issue with the latest master code 2.1-dev2-4a66013. It 
does compile but doesn't want to load my crt-list with .ocsp files 
present for the certificates mentioned. The commit that broke this is: 
246c024


# haproxy -v
HA-Proxy version 2.1-dev2-4a66013 2019/10/14 - https://haproxy.org/
# haproxy -f ./PB-TEST/ultimo_testcase/xxx/haproxy.cfg -d
[ALERT] 286/223026 (39111) : parsing 
[./PB-TEST/ultimo_testcase/xxx/haproxy.cfg:61] : 'bind 0.0.0.0:443' : 
'crt-list' : error processing line 1 in file 
'/usr/ports-pb_haproxy-devel/PB-TEST/ultimo_testcase/xxx/rtrcld.xxx.crt_list' 
: (null)
[ALERT] 286/223026 (39111) : Error(s) found in configuration file : 
./PB-TEST/ultimo_testcase/xxx/haproxy.cfg

[ALERT] 286/223026 (39111) : Fatal errors found in configuration.

Content of the crt-list file, but removing the alpn stuff doesn't help...:
/usr/ports-pb_haproxy-devel/PB-TEST/ultimo_testcase/xxx/rtrcld.xxx.pem [ 
alpn h2,http/1.1]
/usr/ports-pb_haproxy-devel/PB-TEST/ultimo_testcase/xxx/rtrcld.xxx/rtrcld.xxx_5ab0da70ab0cc.pem 
[ alpn h2,http/1.1]


The last line is an empty one.. but it already complains about line 1... 
which seems valid and the .pem file exists.. exact same config loads 
alright commits before this one: 246c024.


I do have a 'filled' .ocsp file present. But no matter if its outdated, 
empty or correct the error above stays. When the .ocsp is absent it 
complains about line 2 of the cert-list.. Which has its own .ocsp as well..


Can you take a look? Thanks in advance.

Regards,
PiBa-NL (Pieter)

Re: freebsd builds are broken for few days - 30ee1ef, proxy_protocol_random_fail.vtc fails because scheme and host are now present in the syslog output.

[PiBa-NL]

Hi Ilya, Willy,

Op 13-10-2019 om 19:30 schreef Илья Шипицин:

https://cirrus-ci.com/github/haproxy/haproxy

I'll bisect if noone else knows what's going on


@IIlya, thanks for checking my favorite platform, FreeBSD ;).

@Willy, this 30ee1ef 
 (MEDIUM: h2: 
use the normalized URI encoding for absolute form requests) commit 
'broke' the expect value of the vtest, i don't know why other platforms 
don't see the same change in syslog output though.. Anyhow this is the 
output i get when running the 
/reg-tests/connection/proxy_protocol_random_fail.vtc


 Slog_1  0.033 syslog|<134>Oct 14 19:56:34 haproxy[78982]: ::1:47040 
[14/Oct/2019:19:56:34.391] ssl-offload-http~ ssl-offload-http/http 
0/0/0/0/0 503 222 - -  1/1/0/0/0 0/0 "POST https://[::1]:47037/1 
HTTP/2.0"
**   Slog_1  0.033 === expect ~ "ssl-offload-http/http .* \"POST /[1-8] 
HTTP/(2\\.0...
 Slog_1  0.033 EXPECT FAILED ~ "ssl-offload-http/http .* "POST 
/[1-8] HTTP/(2\.0|1\.1)""


If i change the vtc vtest file from:
    expect ~ "ssl-offload-http/http .* \"POST /[1-8] HTTP/(2\\.0|1\\.1)\""
To:
    expect ~ "ssl-offload-http/http .* \"POST 
https://[[]::1]:[0-9]{1,5}/[1-8] HTTP/(2\\.0|1\\.1)\""

or:
    expect ~ "ssl-offload-http/http .* \"POST 
https://[[]${h1_ssl_addr}]:${h1_ssl_port}/[1-8] HTTP/(2\\.0|1\\.1)\""


Then the test succeeds for me... but now the question is, should or 
shouldn't the scheme and host be present in the syslog output on all 
platforms.? Or should the regex contain a (optional?) check for this 
extra part? (Also note that even with these added variables in my second 
regext attempt its still using accolades around the IPv6 address.. not 
sure if all machines would use ipv6 for their localhost connection..)


Regards,
PiBa-NL (Pieter)

Re:haproxy.com: Your Website Needs Attention!!

[hallymartin]

Dear haproxy.com Team,


Hope you are doing fine.


I checked your website haproxy.com and wanted to send you a quick note. If  
you want then we can make few changes to place your website on top of  
search engines in the organic search for some selected word phrases.



Your website is currently not performing well on the web market because of  
some of the following issues:



· Due to poor and unauthorized links.
· Relevant keyword phrases are not visible on first page listing.
· Your website is not search engine friendly.
· Website is having on-page and on-site issues.
· No off-page works have been implemented.

Areas of Improvement:


· Get quality content and theme based back links.
· We will give you 1st page ranking on major search engine ie Google.
· Improve your organic traffic and sales.
· Secure your website from Google's latest updates.
· Target your relevant market to increase business.

NOTE: We give guarantee to improve your keyword ranking on Google from the  
first month itself. If we fail to achieve then we will refund your 100%  
money.





Our main objective is to increase your website's online visibility which  
results in improvement in traffic, link popularity, good conversion and ROI.



For more details please reply. We have your WEBSITE ANALYSIS REPORT ready  
with us.



Thanks & Regards,

Hally Martin

Digital Marketing Expert



Disclaimer: We are not spammers. Reply back with us "Interested" and get  
issue fixing plan now only at $199 (One Time)

HAproxy transparent proxy and IPv6

[Philipp Kolmann]

Hi,

I have setup my test-HAproxy-env according to

https://www.haproxy.com/blog/howto-transparent-proxying-and-binding-with-haproxy-and-aloha-load-balancer/

I have setup the Firewall Rules for ipv4 and v6.

TEST testha1:~/svnconfig/etc/iptables# iptables -t mangle -vL
Chain PREROUTING (policy ACCEPT 163K packets, 291M bytes)
 pkts bytes target prot opt in out source destination
 374K   68M DIVERT tcp  --  any    any anywhere 
anywhere socket


Chain DIVERT (1 references)
 pkts bytes target prot opt in out source destination
 374K   68M MARK   all  --  any    any anywhere 
anywhere MARK set 0x1

 374K   68M ACCEPT all  --  any    any anywhere anywhere


TEST testha1:~/svnconfig/etc/iptables# ip6tables -t mangle -vL
Chain PREROUTING (policy ACCEPT 409K packets, 788M bytes)
 pkts bytes target prot opt in out source destination
 373K   75M DIVERT tcp  any    any anywhere 
anywhere socket


Chain DIVERT (1 references)
 pkts bytes target prot opt in out source destination
 373K   75M MARK   all  any    any anywhere 
anywhere MARK set 0x1

 373K   75M ACCEPT all  any    any anywhere anywhere


I have added the required ip cmds:

    post-up ip rule add fwmark 1 lookup 100
    post-up ip route add local 0.0.0.0/0 dev lo table 100
    post-up ip route add local ::/0 dev lo table 100

listen mail-test-submission
    bind 128.130.xx.yy:587 transparent name submission
    mode tcp
    source 0.0.0.0 usesrc clientip
    log-format %ci:%cp\ [%t]\ %ft\ %s\ %si:%sp\ %Tw/%Tc/%Tt\ %B\ 
%ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq

    balance leastconn


That works like a charm.

In IPv6 I set it up accordingly:

listen mail-test-v6-submission
    bind 2001:629:xx:yy::zz:587 transparent name submission
    mode tcp
    source [::] usesrc clientip
    log-format %ci:%cp\ [%t]\ %ft\ %s\ %si:%sp\ %Tw/%Tc/%Tt\ %B\ 
%ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq

    balance leastconn


There with the source line it fails to connect.

I see on the outside interface a Syn, Syn->Ack, Ack TCP flow, but on the 
inside (HAproxy to application Server) I see only Syn, Syn-Ack, Syn, 
Syn-Ack traffic.


HAproxy (1.8.19-1, Debian Buster) is running as root.

Anyone has such a setup running and may be able to help. I haven't found 
any hints on this problem...


Thanks
Philipp

--
---
DI Mag. Philipp Kolmann  mail: philipp.kolm...@tuwien.ac.at
Technische Universitaet Wien   web: www.it.tuwien.ac.at
IT Solutions - Applications  tel: +43(1)58801-42011
Operngasse 11, A-1040 Wien DVR: 0005886
---



smime.p7s
Description: S/MIME Cryptographic Signature