Re: haproxy segfault
On Tue, Feb 12, 2019 at 10:28:01PM +0100, Lukas Tribus wrote: > > Did this bug has been introduced in 1.9.4 ? > > I haven't notice this behavior before. > > Yes. Also see the 1.8.19 release notes. I forgot we backported it to 1.9.4, I thought it was only queued after 1.9.4, we should have emitted a 1.9.5 when doing 1.8.19. I'll probably do one early next week. There's another issue I want to look at quickly before doing this release, I've seen some compression oddities on haproxy.org where some gitweb pages were returned uncompressed with "content-encoding: gzip" being presented. This looks recent, I need to compare with previous stats. Willy
Re: Tune HAProxy in front of a large k8s cluster
On Fri, Feb 15, 2019 at 08:35:58PM -0200, Joao Morais wrote: > This is just theory - and 5 digits starts on 1 =) . The problem is the > " if " multiplied by 3000 or so. > Moving everything to the backend or a map is a really big step forward. I > suspect my frontend will have about 10 lines instead of 4000. The most horrible config I've see had 45 such lines with their respective ACLs. All internet routes were generated for two interfaces to decide which backend to use! The config took some time to load and was not very fast but still it ran fast. In your case I suspect that one of the ACL keywords is expensive to evaluate. I'd be interested in getting a sample of these lines to have an idea. We might very well have a bug in one of these sample fetch functions, or it could be relying on something expensive. If you have some time to run some extra tests, it would be nice to rebuild haproxy with "ARCH_FLAGS=-pg", run it again, stop it using kill -USR1 (not ctrl-C), and run "gprof haproxy gmon.out". It will show the number of calls to each function and a rough approximation of the time spent there. We may find a huge number of calls to the culprit and possibly we could improve it. If it's just that the sample is expensive, there is the option of copying it into a variable and using the variable in the ACL lookups, it will be way cheaper. But of course as Aleks said, using the maps with dynamic backend rules is the right way to proceed in your case. Willy
Re: Tune HAProxy in front of a large k8s cluster
> Em 15 de fev de 2019, à(s) 19:22, Aleksandar Lazic > escreveu: > > Am 15.02.2019 um 22:11 schrieb Joao Morais: >> >> Hey Aleks, this made my day. Thank you for remember me that map exist and a >> big thank you to The Author of map, map_beg and map_reg converters! Time to >> achieve a 5 digits rps. > > Wow you mean from 600 rps to 6 rps "just" to switch to map. > That's amazing 8-O, especially it was just a wild guess ;-) > This is just theory - and 5 digits starts on 1 =) . The problem is the " if " multiplied by 3000 or so. Moving everything to the backend or a map is a really big step forward. I suspect my frontend will have about 10 lines instead of 4000. > Would you like to share your config now? > I’d share but it has some sensitive data. But trust in me, this is as simple as one tcp frontend on :443 with about 100 use_backend -> one http frontend on an unix socket with about 3000 " if” with a few regex -> the backend. I'll let you know as soon as I've updated the config and ran the same tests again ;) ~jm
Re: Tune HAProxy in front of a large k8s cluster
Am 15.02.2019 um 22:11 schrieb Joao Morais: > >> Em 15 de fev de 2019, à(s) 08:43, Aleksandar Lazic >> escreveu: >> >> Hi Joao. >> >> Am 15.02.2019 um 11:15 schrieb Joao Morais: >>> >>> Hi Aleks, sure. Regarding the config, it has currently about 4k lines >>> only in the largest frontend because of the number of hostnames and paths >>> being supported. About 98% is acl declarations, http-request, reqrep, >>> redirect scheme, use_backend. Most of them I'll move to the backend and >>> this will already improve performance. The question is: what about the >>> 2200+ `use_backend` - is there anything else that could be done? >> >> As I don't know the config, even a snippet could help, let me suggest you >> to try to use a map for lookup for the backends. >> >> https://www.haproxy.com/blog/introduction-to-haproxy-maps/ >> > Hey Aleks, this made my day. Thank you for remember me that map exist and a > big thank you to The Author of map, map_beg and map_reg converters! Time to > achieve a 5 digits rps. Wow you mean from 600 rps to 6 rps "just" to switch to map. That's amazing 8-O, especially it was just a wild guess ;-) Would you like to share your config now? >> Do you use DNS resolving for the hostnames? >> > Nops. My guess so far is the size of the frontend and my tests have confirmed > this. > >>> / # haproxy -vv HA-Proxy version 1.8.17 2019/01/08 >> >> Event it's not critical, it would be nice when you can try 1.8.19 or >> better 1.9.4 ;-) >> > Moving to 1.8.19 soon and waiting 1.9.5 =) ;-) > ~jm Regards Aleks
Re: Tune HAProxy in front of a large k8s cluster
> Em 15 de fev de 2019, à(s) 08:43, Aleksandar Lazic > escreveu: > > Hi Joao. > > Am 15.02.2019 um 11:15 schrieb Joao Morais: >> >> Hi Aleks, sure. Regarding the config, it has currently about 4k lines only >> in the largest frontend because of the number of hostnames and paths being >> supported. About 98% is acl declarations, http-request, reqrep, redirect >> scheme, use_backend. Most of them I'll move to the backend and this will >> already improve performance. The question is: what about the 2200+ >> `use_backend` - is there anything else that could be done? > > As I don't know the config, even a snippet could help, let me suggest you to > try > to use a map for lookup for the backends. > > https://www.haproxy.com/blog/introduction-to-haproxy-maps/ > Hey Aleks, this made my day. Thank you for remember me that map exist and a big thank you to The Author of map, map_beg and map_reg converters! Time to achieve a 5 digits rps. > Do you use DNS resolving for the hostnames? > Nops. My guess so far is the size of the frontend and my tests have confirmed this. >> / # haproxy -vv >> HA-Proxy version 1.8.17 2019/01/08 > > Event it's not critical, it would be nice when you can try 1.8.19 or better > 1.9.4 ;-) > Moving to 1.8.19 soon and waiting 1.9.5 =) ~jm
Re: HAProxy in front of Docker Enterprise problem
> Em 12 de fev de 2019, à(s) 21:21, Norman Branitsky > escreveu: > > Do I have to make HAProxy listen on 8443 and just do a tcp frontend/backend > for the Manager nodes? You can bind on another port, you can also bind on another IP address (change *:443 to some.ip.addr:443). But if you want or you need to share the same IP and port, a possible configuration is to create a tcp mode frontend which inspect sni extension and make a triage: manager hostname? Use a tcp mode backend and the manager nodes as servers - no data would be changed. This blog post[1] is of some help. In the triage, if the request isn't to a maanger node, use another tcp backend whose only server is a unix socket. Use also send-proxy-v2 in the server declaration. Create another http mode frontend, binding that unix socket and accept-proxy keyword to do the ssl offload of your worker nodes. hth. [1] https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension/
Re: Tune HAProxy in front of a large k8s cluster
Hi Joao. Am 15.02.2019 um 11:15 schrieb Joao Morais: > > >> Em 15 de fev de 2019, à(s) 07:44, Aleksandar Lazic >> escreveu: >> >> Hi Joao. >> >> Am 15.02.2019 um 10:21 schrieb Joao Morais: >>> >>> Hi list, I'm tuning some HAProxy instances in front of a large kubernetes >>> cluster. The config has about 500 hostnames (a la apache/nginx virtual >>> hosts), 3 frontends, 1500 backends and 4000 servers. The first frontend is >>> on >>> tcp mode binding :443, inspecting sni and doing a triage; the second >>> frontend >>> is binding a unix socket with ca-file (tls authentication); the last >>> frontend >>> is binding another unix socket, doing ssl-offload but without ca-file. This >>> last one has about 80% of the hostnames. There is also a ssl-passthrough >>> config - from the triage frontend straight to a tcp backend. >> >> Please can you tell us which haproxy you use and show us the config, thanks. > > Hi Aleks, sure. Regarding the config, it has currently about 4k lines only in > the largest frontend because of the number of hostnames and paths being > supported. About 98% is acl declarations, http-request, reqrep, redirect > scheme, use_backend. Most of them I'll move to the backend and this will > already improve performance. The question is: what about the 2200+ > `use_backend` - is there anything else that could be done? As I don't know the config, even a snippet could help, let me suggest you to try to use a map for lookup for the backends. https://www.haproxy.com/blog/introduction-to-haproxy-maps/ Do you use DNS resolving for the hostnames? > / # haproxy -vv > HA-Proxy version 1.8.17 2019/01/08 Event it's not critical, it would be nice when you can try 1.8.19 or better 1.9.4 ;-) Regards Aleks
Re: Tune HAProxy in front of a large k8s cluster
> Em 15 de fev de 2019, à(s) 07:44, Aleksandar Lazic
> escreveu:
>
> Hi Joao.
>
> Am 15.02.2019 um 10:21 schrieb Joao Morais:
>>
>> Hi list, I'm tuning some HAProxy instances in front of a large kubernetes
>> cluster. The config has about 500 hostnames (a la apache/nginx virtual
>> hosts), 3 frontends, 1500 backends and 4000 servers. The first frontend is on
>> tcp mode binding :443, inspecting sni and doing a triage; the second frontend
>> is binding a unix socket with ca-file (tls authentication); the last frontend
>> is binding another unix socket, doing ssl-offload but without ca-file. This
>> last one has about 80% of the hostnames. There is also a ssl-passthrough
>> config - from the triage frontend straight to a tcp backend.
>
> Please can you tell us which haproxy you use and show us the config, thanks.
Hi Aleks, sure. Regarding the config, it has currently about 4k lines only in
the largest frontend because of the number of hostnames and paths being
supported. About 98% is acl declarations, http-request, reqrep, redirect
scheme, use_backend. Most of them I'll move to the backend and this will
already improve performance. The question is: what about the 2200+
`use_backend` - is there anything else that could be done?
/ # haproxy -vv
HA-Proxy version 1.8.17 2019/01/08
Copyright 2000-2019 Willy Tarreau
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
-Wno-null-dereference -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2q 20 Nov 2018
Running on OpenSSL version : OpenSSL 1.0.2q 20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
Re: Tune HAProxy in front of a large k8s cluster
Hi Joao. Am 15.02.2019 um 10:21 schrieb Joao Morais: > > Hi list, I'm tuning some HAProxy instances in front of a large kubernetes > cluster. The config has about 500 hostnames (a la apache/nginx virtual > hosts), 3 frontends, 1500 backends and 4000 servers. The first frontend is on > tcp mode binding :443, inspecting sni and doing a triage; the second frontend > is binding a unix socket with ca-file (tls authentication); the last frontend > is binding another unix socket, doing ssl-offload but without ca-file. This > last one has about 80% of the hostnames. There is also a ssl-passthrough > config - from the triage frontend straight to a tcp backend. Please can you tell us which haproxy you use and show us the config, thanks. haproxy -vv Regars Aleks > I'm observing some latency on moderate loads (200+ rps per instance) - on my > tests, the p95 was about 25ms only in the proxy, and the major issue is that > I cannot have a throughput above 600 rps. This latency moves easily from 25ms > on p95 to 1s or more on p50 with 700+ rps. The problem is of course the big > amount of rules in the frontend: haproxy need to check every single bit of > configuration for every single host and every single path. Moving the testing > hostname to a dedicated frontend with only its own rules give me with about > 5ms of p95 latency and more than 5000 rps. > > These are my ideas so far regarding tune such configuration: > > * Move all possible rules to the backend. Some txn vars should be created in > order to be inspected there. This will of course help but there is still a > lot of `use_backend if ` that cannot be removed, I > think, which are being evaluated on every single request despite the hostname > that I'm really interested. There are some hostnames without path acl, but > there are also hostnames with 10+ different paths and its 10+ `use_backend`. > > * Create some more frontends and unix sockets with at most 50 hostnames or > so. Pros: after the triage, each frontend will have the `use_backend if` of > only another 49 hostnames. Cons: if some client doesn't send the sni > extension, the right frontend couldn't be found. > > * Perhaps there is a hidden `if do done` that I'm > missing which would improve performance, since I can help HAProxy to process > only the keywords I'm really interested in that request. > > * Nbthreads was already tested, I'm using 3 that has the best performance on > a 8 cores VM. 4+ threads doesn’t scale. Nbprocs will also be used, I'm tuning > a per process configuration now. > > Is there any other approach I'm missing? Every single milisecond will help. >
Tune HAProxy in front of a large k8s cluster
Hi list, I'm tuning some HAProxy instances in front of a large kubernetes cluster. The config has about 500 hostnames (a la apache/nginx virtual hosts), 3 frontends, 1500 backends and 4000 servers. The first frontend is on tcp mode binding :443, inspecting sni and doing a triage; the second frontend is binding a unix socket with ca-file (tls authentication); the last frontend is binding another unix socket, doing ssl-offload but without ca-file. This last one has about 80% of the hostnames. There is also a ssl-passthrough config - from the triage frontend straight to a tcp backend. I'm observing some latency on moderate loads (200+ rps per instance) - on my tests, the p95 was about 25ms only in the proxy, and the major issue is that I cannot have a throughput above 600 rps. This latency moves easily from 25ms on p95 to 1s or more on p50 with 700+ rps. The problem is of course the big amount of rules in the frontend: haproxy need to check every single bit of configuration for every single host and every single path. Moving the testing hostname to a dedicated frontend with only its own rules give me with about 5ms of p95 latency and more than 5000 rps. These are my ideas so far regarding tune such configuration: * Move all possible rules to the backend. Some txn vars should be created in order to be inspected there. This will of course help but there is still a lot of `use_backend if ` that cannot be removed, I think, which are being evaluated on every single request despite the hostname that I'm really interested. There are some hostnames without path acl, but there are also hostnames with 10+ different paths and its 10+ `use_backend`. * Create some more frontends and unix sockets with at most 50 hostnames or so. Pros: after the triage, each frontend will have the `use_backend if` of only another 49 hostnames. Cons: if some client doesn't send the sni extension, the right frontend couldn't be found. * Perhaps there is a hidden `if do done` that I'm missing which would improve performance, since I can help HAProxy to process only the keywords I'm really interested in that request. * Nbthreads was already tested, I'm using 3 that has the best performance on a 8 cores VM. 4+ threads doesn’t scale. Nbprocs will also be used, I'm tuning a per process configuration now. Is there any other approach I'm missing? Every single milisecond will help.
Re: Early connection close, incomplete transfers
Am 15.02.2019 um 08:47 schrieb Veiko Kukk: > On 2019-02-14 18:29, Aleksandar Lazic wrote: >>> Replaced HAproxy with Nginx for testing and with Nginx, not a single >>> connection >>> was interrupted, did millions of requests. >> >> In 1.9.4 are a lot of fixed added. >> please can you try your tests with 1.9.4, thanks. > > Already did before writing my previous letter. No differencies. Okay. Can you try to make some tcpdumps between haproxy and lighty, we need to know what happens on the wire. > Veiko Regards Aleks
