Re: Few problems seen in haproxy? (threads, connections).
Hi Krishna, On Fri, Oct 05, 2018 at 02:25:13PM +0530, Krishna Kumar (Engineering) wrote: > Sorry for repeating once again, but this is my last unsolicited > mail on this topic. Any directions for what to look out for? Sorry, but I didn't even have the time to read your mail over the last two days and still have a huge backlog pending. Maybe next week. Regards, Willy
Re: lua haproxy-auth-request - round 2
Hi Tim,
Thanks for the response, and apologies for the delay. Popularity is
advertised as a good thing, but I have my doubts. Regardless, I am back
on track with this project for the moment.
While I don't have any idea from the top of my head your configuration
surely would be helpful.
right, should have included that the first time, will put it at the
bottom of this mail.
You might also want to check whether the webbrowser is able to:
a) Set-Up the Websocket with auth-request in between (you should see a
101 Switching Protocols in it's network console).
Okay, this is something I hadn't looked at. Not 100% sure I am
interpreting correctly, but assuming I am, then the browser is reporting
that the connection is successfully upgrading to websockets both with
and without the Lua script enabled. As far as I can tell, the only
thing that changes is the cookies and keys. There is a significant
difference in the "waiting" response, though; ~350ms with the lua
script, but less than 5ms without it.
b) Send credentials for basic authentication for Websockets.
hm, so I don't seem to be able to connect to etherpad directly using a
ws:// schematic in chrome or firefox, and I think that is what you mean.
not sure if that is just me, though, will work some more on that and
see if I can figure out if I am doing something wrong there...
My haproxy.cfg:
Note in the front end I have two lines commented; with these lines
commented, everything works, can reload hundreds of times with no error.
With the lines uncommented, the auth works, the main landing page
works, but accessing the actual pad does not work. the tables entries
seem to work fine either way.
I have uncommented these lines to induce failure, and pasted a copy of
the haproxy logs of the event at
http://www.computerisms.ca/haproxy.txt
At the bottom of the log file, it appears to me that I get a 200 after
the websocket upgrade, which I interpret to mean it was successful, but
at that point the page spins and some 150-300 seconds later I get the
error page displayed on the screen with no more log entries in haproxy.
global
debug
log /dev/loglocal1 debug
chroot /var/lib/haproxy
userhaproxy
group haproxy
daemon
ca-base /etc/ssl/certs
crt-base/etc/ssl/private
ssl-default-bind-optionsno-sslv3
lua-load/Computerisms/config/etc/haproxy.auth.lua
defaults
log global
modehttp
option httplog
option dontlognull
frontendhttpfront
bind${ADDRESS}:80 v4v6
bind ${ADDRESS}:443 v4v6 ssl crt /Computerisms/config/certificates/
redirectscheme https code 301 if !{ ssl_fc }
modehttp
option httplog
log global
# http-requestlua.auth-requestauth_request /index.html
## ACLs
acl tables.computerisms.ca ssl_fc_sni -i tables.computerisms.ca
acl pad.computerisms.ca ssl_fc_sni -i pad.computerisms.ca
## AUTHREQ
use_backend auth_request if ! {
var(txn.auth_response_successful) -m bool } tables.computerisms.ca
# use_backend auth_request if ! {
var(txn.auth_response_successful) -m bool } pad.computerisms.ca
## AUTHBACKEND
use_backend tables.computerisms.ca if tables.computerisms.ca
use_backend pad.computerisms.ca if pad.computerisms.ca
default_backend mooglehttps
backend auth_request
modehttp
server auth-request127.0.0.1:8044 check
# option httpclose
# option forwardfor
backend mooglehttps
balance leastconn
modehttp
option httpclose
option forwardfor
option log-health-checks
option httpchk
server sand1lian 192.168.25.52:48443 check send-proxy-v2 ssl
verify none
server sand2lian 192.168.25.53:48443 check send-proxy-v2 ssl
verify none
## BEGIN pad.computerisms.ca
backend pad.computerisms.ca
balance leastconn
modehttp
cookie sessionID insert nocache indirect
#option httpclose
option forwardfor
server sand1lian 192.168.25.52:19008 cookie sand1pad
server sand2lian 192.168.25.53:19008 cookie sand2pad
## END pad.computerisms.ca
## BEGIN tables.computerisms.ca
backend tables.computerisms.ca
balance leastconn
modehttp
cookie sessionID insert nocache indirect
option httpclose
option forwardfor
server sand1lian 192.168.25.52:29000 check cookie sand1tables
server sand2lian 192.168.25.53:29000 check
Re: Combine different ACLs under same name
El vie, 05-10-2018 a las 11:38 +0200, Jerome Magnin escribió:
> Hello,
>
> On Fri, Oct 05, 2018 at 10:46:20AM +0200, Ricardo Fraile wrote:
> > Hello,
> >
> >
> > I have tested that some types of acls can't be combined, as example:
> >
> > Server 192.138.1.1, acl with combined rules:
> >
> > acl rule1 hdr_dom(host) -i test.com
> > acl rule1 src 192.168.1.2/24
> > redirect prefix
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyes.comdata=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=8RT5c2eXB%2FFk6TDNe6TqXyDmy8YRgVpSz2WbjXggFCg%3Dreserved=0
> > code 301 if rule1
> > redirect prefix
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fno.comdata=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=Rt4XuK0X7D81dEQ9aNyviySqJInlLQg1U%2BdGX%2BBCtcM%3Dreserved=0
> >
> > Request from 192.168.1.2:
> >
> > $ curl -I -H "host: test.com" 192.138.1.1
> > HTTP/1.1 301 Moved Permanently
> > Content-length: 0
> > Location:
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyes.com%2Fdata=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=xLRo6a963KFqYn7BSmtUSb96EI7rLLuyVSwyfcdfP%2Bo%3Dreserved=0
> >
> > Request from 192.168.1.3:
> >
> > $ curl -I -H "host: test.com" 192.138.1.1
> > HTTP/1.1 301 Moved Permanently
> > Content-length: 0
> > Location:
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyes.com%2Fdata=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=xLRo6a963KFqYn7BSmtUSb96EI7rLLuyVSwyfcdfP%2Bo%3Dreserved=0
> >
> >
> >
> > Server 192.138.1.1, acl with two rules:
> >
> > acl rule1 hdr_dom(host) -i test.com
> > acl rule2 src 192.168.1.2/24
> > redirect prefix
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyes.comdata=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=8RT5c2eXB%2FFk6TDNe6TqXyDmy8YRgVpSz2WbjXggFCg%3Dreserved=0
> > code 301 if rule1 rule2
> > redirect prefix
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fno.comdata=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=Rt4XuK0X7D81dEQ9aNyviySqJInlLQg1U%2BdGX%2BBCtcM%3Dreserved=0
> >
> > Request from 192.168.1.2:
> >
> > $ curl -I -H "host: test.com" 192.138.1.1
> > HTTP/1.1 301 Moved Permanently
> > Content-length: 0
> > Location:
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyes.com%2Fdata=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=xLRo6a963KFqYn7BSmtUSb96EI7rLLuyVSwyfcdfP%2Bo%3Dreserved=0
> >
> > Request from 192.168.1.3:
> >
> > $ curl -I -H "host: test.com" 192.138.1.1
> > HTTP/1.1 301 Moved Permanently
> > Content-length: 0
> > Location:
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fno.com%2Fdata=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=8oG7jYs129GAJb9uqBZOp0c09KqCG6gLsR%2FctUsFsfM%3Dreserved=0
> >
> > I look for this behaviour on the documentation but I don't find any
> > reference to it. Please, can someone know where it is documented?
> >
> >
>
> This is expected behavior.
>
> when you declare acls with the same name such as:
>
> acl foo src 1.2.3.4
> acl foo hdr(host) foo.bar
>
>
> and use foo as a condition for anything, foo equivalent to :
>
> { src 1.2.3.4 } || { hdr(host) foo.bar }
>
> There is at least an example of this behavior in the documentation:
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcbonte.github.io%2Fhaproxy-dconv%2F1.8%2Fconfiguration.html%237.2data=02%7C01%7C%7C0a6e0b206dd5474eaeee08d62aa6535d%7Cd78b7929c2a34897ae9a7d8f8dc1a1cf%7C0%7C0%7C636743291183969700sdata=q%2BDgDSduhH6PoH43SEG0VA4Ywesrs%2FP4EtYVpBMc4m4%3Dreserved=0
>
> your splitting of the acl in two acls leads to implying an && between the two
> acls, and the behavior is different.
>
> regards,
> Jérôme
It is definitely clever, indeed.
If it is possible, as suggestion, I think that it need to be more clear
on the documentation.
Thanks,
Re: Combine different ACLs under same name
Hello,
On Fri, Oct 05, 2018 at 10:46:20AM +0200, Ricardo Fraile wrote:
> Hello,
>
>
> I have tested that some types of acls can't be combined, as example:
>
> Server 192.138.1.1, acl with combined rules:
>
> acl rule1 hdr_dom(host) -i test.com
> acl rule1 src 192.168.1.2/24
> redirect prefix https://yes.com code 301 if rule1
> redirect prefix https://no.com
>
> Request from 192.168.1.2:
>
> $ curl -I -H "host: test.com" 192.138.1.1
> HTTP/1.1 301 Moved Permanently
> Content-length: 0
> Location: https://yes.com/
>
> Request from 192.168.1.3:
>
> $ curl -I -H "host: test.com" 192.138.1.1
> HTTP/1.1 301 Moved Permanently
> Content-length: 0
> Location: https://yes.com/
>
>
>
> Server 192.138.1.1, acl with two rules:
>
> acl rule1 hdr_dom(host) -i test.com
> acl rule2 src 192.168.1.2/24
> redirect prefix https://yes.com code 301 if rule1 rule2
> redirect prefix https://no.com
>
> Request from 192.168.1.2:
>
> $ curl -I -H "host: test.com" 192.138.1.1
> HTTP/1.1 301 Moved Permanently
> Content-length: 0
> Location: https://yes.com/
>
> Request from 192.168.1.3:
>
> $ curl -I -H "host: test.com" 192.138.1.1
> HTTP/1.1 301 Moved Permanently
> Content-length: 0
> Location: https://no.com/
>
> I look for this behaviour on the documentation but I don't find any
> reference to it. Please, can someone know where it is documented?
>
>
This is expected behavior.
when you declare acls with the same name such as:
acl foo src 1.2.3.4
acl foo hdr(host) foo.bar
and use foo as a condition for anything, foo equivalent to :
{ src 1.2.3.4 } || { hdr(host) foo.bar }
There is at least an example of this behavior in the documentation:
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.2
your splitting of the acl in two acls leads to implying an && between the two
acls, and the behavior is different.
regards,
Jérôme
Re: Few problems seen in haproxy? (threads, connections).
Sorry for repeating once again, but this is my last unsolicited
mail on this topic. Any directions for what to look out for?
Thanks,
- Krishna
On Thu, Oct 4, 2018 at 8:42 AM Krishna Kumar (Engineering) <
krishna...@flipkart.com> wrote:
> Re-sending in case this mail was missed. To summarise the 3 issues seen:
>
> 1. Performance drops 18x with higher number of nbthreads as compared to
> nbprocs.
> 2. CPU utilisation remains at 100% after wrk finishes for 30 seconds (for
> 1.9-dev3
> for nbprocs and nbthreads).
> 3. Sockets on client remain in FIN-WAIT-2, while on HAProxy it remains in
> either
> CLOSE-WAIT (towards clients) and ESTAB (towards the backend servers),
> till
> the server/client timeout expires.
>
> The tests for threads and processes were done on the same systems, so
> there is
> no difference in system parameters.
>
> Thanks,
> - Krishna
>
>
> On Tue, Oct 2, 2018 at 9:18 PM Krishna Kumar (Engineering) <
> krishna...@flipkart.com> wrote:
>
>> Hi Willy, and community developers,
>>
>> I am not sure if I am doing something wrong, but wanted to report
>> some issues that I am seeing. Please let me know if this is a problem.
>>
>> 1. HAProxy system:
>> Kernel: 4.17.13,
>> CPU: 48 core E5-2670 v3
>> Memory: 128GB memory
>> NIC: Mellanox 40g with IRQ pinning
>>
>> 2. Client, 48 core similar to server. Test command line:
>> wrk -c 4800 -t 48 -d 30s http:///128
>>
>> 3. HAProxy version: I am testing both 1.8.14 and 1.9-dev3 (git checkout
>> as of
>> Oct 2nd).
>> # haproxy-git -vv
>> HA-Proxy version 1.9-dev3 2018/09/29
>> Copyright 2000-2018 Willy Tarreau
>>
>> Build options :
>> TARGET = linux2628
>> CPU = generic
>> CC = gcc
>> CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
>> -fwrapv -fno-strict-overflow -Wno-unused-label -Wno-sign-compare
>> -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers
>> -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
>> OPTIONS = USE_ZLIB=yes USE_OPENSSL=1 USE_PCRE=1
>>
>> Default settings :
>> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>>
>> Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
>> Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
>> OpenSSL library supports TLS extensions : yes
>> OpenSSL library supports SNI : yes
>> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
>> Built with transparent proxy support using: IP_TRANSPARENT
>> IPV6_TRANSPARENT IP_FREEBIND
>> Encrypted password support via crypt(3): yes
>> Built with multi-threading support.
>> Built with PCRE version : 8.38 2015-11-23
>> Running on PCRE version : 8.38 2015-11-23
>> PCRE library supports JIT : no (USE_PCRE_JIT not set)
>> Built with zlib version : 1.2.8
>> Running on zlib version : 1.2.8
>> Compression algorithms supported : identity("identity"),
>> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
>> Built with network namespace support.
>>
>> Available polling systems :
>> epoll : pref=300, test result OK
>>poll : pref=200, test result OK
>> select : pref=150, test result OK
>> Total: 3 (3 usable), will use epoll.
>>
>> Available multiplexer protocols :
>> (protocols markes as cannot be specified using 'proto' keyword)
>> h2 : mode=HTTP side=FE
>> : mode=TCP|HTTP side=FE|BE
>>
>> Available filters :
>> [SPOE] spoe
>> [COMP] compression
>> [TRACE] trace
>>
>> 4. HAProxy results for #processes and #threads
>> #Threads-RPS Procs-RPS
>> 1 20903 19280
>> 2 46400 51045
>> 4 96587 142801
>> 8 172224 254720
>> 12 210451 437488
>> 16 173034 437375
>> 24 79069 519367
>> 32 55607 586367
>> 48 31739 596148
>>
>> 5. Lock stats for 1.9-dev3: Some write locks on average took a lot more
>> time
>>to acquire, e.g. "POOL" and "TASK_WQ". For 48 threads, I get:
>> Stats about Lock FD:
>> # write lock : 143933900
>> # write unlock: 143933895 (-5)
>> # wait time for write : 11370.245 msec
>> # wait time for write/lock: 78.996 nsec
>> # read lock : 0
>> # read unlock : 0 (0)
>> # wait time for read : 0.000 msec
>> # wait time for read/lock : 0.000 nsec
>> Stats about Lock TASK_RQ:
>> # write lock : 2062874
>> # write unlock: 2062875 (1)
>> # wait time for write : 7820.234 msec
>> # wait time for write/lock: 3790.941 nsec
>> # read lock : 0
>> # read unlock : 0 (0)
>> # wait time for read : 0.000 msec
>> # wait time for read/lock : 0.000 nsec
>> Stats about Lock TASK_WQ:
>> # write lock : 2601227
>> # write unlock: 2601227 (0)
>> # wait time for write : 5019.811 msec
>> # wait time for write/lock: 1929.786 nsec
>> # read lock : 0
>> # read unlock : 0 (0)
>> # wait time for read : 0.000 msec
>> # wait time for read/lock : 0.000 nsec
>> Stats about Lock POOL:
>> # write lock : 2823393
>> # write unlock: 2823393 (0)
>> # wait time for write : 11984.706 msec
>> # wait time for write/lock: 4244.788 nsec
>> # read lock : 0
>> # read unlock : 0 (0)
>> # wait time for
Combine different ACLs under same name
Hello, I have tested that some types of acls can't be combined, as example: Server 192.138.1.1, acl with combined rules: acl rule1 hdr_dom(host) -i test.com acl rule1 src 192.168.1.2/24 redirect prefix https://yes.com code 301 if rule1 redirect prefix https://no.com Request from 192.168.1.2: $ curl -I -H "host: test.com" 192.138.1.1 HTTP/1.1 301 Moved Permanently Content-length: 0 Location: https://yes.com/ Request from 192.168.1.3: $ curl -I -H "host: test.com" 192.138.1.1 HTTP/1.1 301 Moved Permanently Content-length: 0 Location: https://yes.com/ Server 192.138.1.1, acl with two rules: acl rule1 hdr_dom(host) -i test.com acl rule2 src 192.168.1.2/24 redirect prefix https://yes.com code 301 if rule1 rule2 redirect prefix https://no.com Request from 192.168.1.2: $ curl -I -H "host: test.com" 192.138.1.1 HTTP/1.1 301 Moved Permanently Content-length: 0 Location: https://yes.com/ Request from 192.168.1.3: $ curl -I -H "host: test.com" 192.138.1.1 HTTP/1.1 301 Moved Permanently Content-length: 0 Location: https://no.com/ I look for this behaviour on the documentation but I don't find any reference to it. Please, can someone know where it is documented? Thanks,
Compression disabling on chunked response
Hi, I see this in the documentation: Compression is disabled when: * ... * response header "Transfer-Encoding" contains "chunked" (Temporary Workaround) * Is this still accurate? I have tested a lot of responses from Server with compression enabled in backend and server sending chunked response, haproxy is compressing the stream correctly. What am I missing? I am trying to figure out in what cases could haproxy not compress a response from server.
