Invalid blank line in master socket output when run in non-admin level

[Daniel MacDougall]

Hi,

I ran across what seems like a bug in the master socket output. When run in
a non-admin level (i.e., operator or user), it outputs an extra newline
before printing the normal output. This seems to break the following
promise from the management guide
:

Since multiple commands may be issued at once, haproxy uses the empty line
as a
delimiter to mark an end of output for each command, and takes care of
ensuring
that no command can emit an empty line on output. A script can thus easily
parse the output even when multiple commands were pipelined on a single
line.


Here's a brief demonstration:

*# At admin level, the output is normal.*
> sudo haproxy -W -S /run/haproxy/master.sock,mode,666,level,*admin* -f
haproxy.cfg
> socat /run/haproxy/master.sock stdio
show cli level
admin

@1 show info
Name: HAProxy
Version: 2.0.1-1ppa1~bionic
Release_date: 2019/06/27
Nbthread: 8
Nbproc: 1
...

*# At operator or user level, the output is preceded by an extra blank
line.*
> sudo haproxy -W -S /run/haproxy/master.sock,mode,666,level,*operator* -f
haproxy.cfg
> socat /run/haproxy/master.sock stdio
show cli level

operator

@1 show info

Name: HAProxy
Version: 2.0.1-1ppa1~bionic
Release_date: 2019/06/27
Nbthread: 8
Nbproc: 1
...

Note that this behavior is only present on the master socket, not on worker
sockets.

For completeness, this is the haproxy.cfg file I'm using:

global
log stdout  format raw  daemon  info

stats socket /run/haproxy/worker.sock mode 666 level operator

chroot /var/lib/haproxy
user haproxy
group haproxy

Is this indeed a bug, or am I missing something? If it is a bug, would it
be appropriate to file an issue on GitHub?

Thanks!
Daniel

fullconn not working

[Patrick Hemmer]

I'm trying to get fullconn working, and can't seem to do so. I dunno if 
it's a bug, or if it's my understanding that's wrong.
Basically my goal is to prevent the cumulative total of all connections 
to all servers in a pool from exceeding a certain value.
For example I might have 10 servers, each with a maxconn of 10. But I 
want to configure haproxy with a pool-wide limit of 50, so that even if 
the connections are well distributed and no one server is maxed out, 
after 50 connections to all servers, haproxy will still start to queue 
instead.


fullconn seems like the right way to accomplish this, however I cannot 
get it to work. I've tried a simple setup of 2 servers, each with 
`maxconn 3`, and then a backend `fullconn 2`, which should result in 
queuing after 2 simultaneous connections, however it doesn't. If I send 
4 connections, all 4 are simultaneously sent to the backend servers.


Here's my test config:
defaults
    log 127.0.0.1:1234 daemon
    mode http
    option httplog
    timeout queue 5s
frontend f1
    bind :8001
    default_backend b1
backend b1
    fullconn 2
    server s1 127.0.0.1:8081 minconn 1 maxconn 3
    server s2 127.0.0.1:8081 minconn 1 maxconn 3

Here's how I test:
for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done

And here's the logs:
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -  
4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 
[28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -  
4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -  
4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
<30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 
[28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -  
4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"



So am I misunderstanding how fullconn works? Or is there a bug?
I've tested with 2.0.1, 1.9.8, and 1.8.13.

-Patrick

Re: haproxy=2.0.1: socket leak

[Olivier Houchard]

Hi Maksim,

On Fri, Jun 28, 2019 at 04:09:48PM +0300, Максим Куприянов wrote:
> Hi!
> 
> I found out that in some situations under high rate of incoming connections
> haproxy=2.0.1 starts leaking sockets. It looks like haproxy doesn't close
> connections to its backends after request is finished (FIN received from
> client) thus leaving its server-sockets in close-wait state.
> 
> As an example this simple config starts leaking right from the start for
> me. And everything ends with messages: "local0.emerg Proxy
> server.local:18400 reached process FD limit (maxsock=4130). Please check
> 'ulimit-n' and restart."
> 
> My config look very similar to this:
> global
>   daemon
>   uid 120
>   gid 126
>   stats socket /var/run/haproxy.sock mode 700 level admin expose-fd
> listeners
>   master-worker
>   log 127.0.0.1:516 local0 warning
>   maxconn 2000
>   tune.ssl.default-dh-param 2048
> 
> defaults
>   log global
>   maxconn 4096
>   modetcp
>   retries 3
>   timeout client  1h
>   timeout connect 5s
>   timeout server  1h
>   option  redispatch
>   option  dontlognull
> 
> listen server.local:18400
>   bind ipv6@::1:18400 tfo
>   bind ipv4@127.0.0.1:18400 tfo
>   mode tcp
>   balance leastconn
> 
>   timeout server 24h
>   timeout client 24h
>   option  dontlog-normal
>   log 127.0.0.1:516 local1 info
>   option  httpchk GET /check HTTP/1.1\r\nHost:\ server.local
>   http-check send-state
>   http-check expect status 200
>   tcp-request inspect-delay 10s
>   tcp-request content reject if { nbsrv lt 1 }
>   default-server weight 50
>   server backend-server.local:17995 backend-server.local:17995 check port
> 17994
> 

What kind of requests do you do ? GET ? POST ? Others ?

Thanks !

Olivier

Re: haproxy=2.0.1: socket leak

[Tim Düsterhus]

Maksim,

Am 28.06.19 um 15:09 schrieb Максим Куприянов:
> Hi!
> 
> I found out that in some situations under high rate of incoming connections
> haproxy=2.0.1 starts leaking sockets. It looks like haproxy doesn't close
> connections to its backends after request is finished (FIN received from
> client) thus leaving its server-sockets in close-wait state.

I believe your report might be a duplicate of this issue in the
bugtracker on GitHub, can you check:
https://github.com/haproxy/haproxy/issues/136

Best regards
Tim Düsterhus

haproxy=2.0.1: socket leak

[Максим Куприянов]

Hi!

I found out that in some situations under high rate of incoming connections
haproxy=2.0.1 starts leaking sockets. It looks like haproxy doesn't close
connections to its backends after request is finished (FIN received from
client) thus leaving its server-sockets in close-wait state.

As an example this simple config starts leaking right from the start for
me. And everything ends with messages: "local0.emerg Proxy
server.local:18400 reached process FD limit (maxsock=4130). Please check
'ulimit-n' and restart."

My config look very similar to this:
global
  daemon
  uid 120
  gid 126
  stats socket /var/run/haproxy.sock mode 700 level admin expose-fd
listeners
  master-worker
  log 127.0.0.1:516 local0 warning
  maxconn 2000
  tune.ssl.default-dh-param 2048

defaults
  log global
  maxconn 4096
  modetcp
  retries 3
  timeout client  1h
  timeout connect 5s
  timeout server  1h
  option  redispatch
  option  dontlognull

listen server.local:18400
  bind ipv6@::1:18400 tfo
  bind ipv4@127.0.0.1:18400 tfo
  mode tcp
  balance leastconn

  timeout server 24h
  timeout client 24h
  option  dontlog-normal
  log 127.0.0.1:516 local1 info
  option  httpchk GET /check HTTP/1.1\r\nHost:\ server.local
  http-check send-state
  http-check expect status 200
  tcp-request inspect-delay 10s
  tcp-request content reject if { nbsrv lt 1 }
  default-server weight 50
  server backend-server.local:17995 backend-server.local:17995 check port
17994

# haproxy -vvv
HA-Proxy version 2.0.1-1 2019/06/27 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-strict-aliasing
-Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers
-Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_GETADDRINFO=1
USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_NS=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE
+PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
+REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
+LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
-MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS
-51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=32).
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.31 2012-07-06
Running on PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (libpcre build without JIT?)
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
  h2 : mode=HTXside=FE|BE mux=H2
  h2 : mode=HTTP   side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP   side=FE|BE mux=PASS

Available services :
prometheus-exporter

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace


--
Best regards,
Maksim Kupriyanov

Re: Config Segmentation Fault [2.0.1]

[Luke Seelenbinder]

Hi Olivier,

That makes sense. I figured it was one of my various odd settings not being 
tested with the other (this config is rather…complex), and I hoped your eyes 
would be better than mine. Glad they were!

Thanks for getting this fixed up. I'll pull the latest git when I have the 
chance and confirm it fixes it.

Best,
Luke
—
Luke Seelenbinder
Stadia Maps | Founder
stadiamaps.com

> On Jun 28, 2019, at 14:14, Olivier Houchard  wrote:
> 
> Hi Luke,
> 
> On Fri, Jun 28, 2019 at 07:05:32AM +0200, Luke Seelenbinder wrote:
>> Hello all,
>> 
>> I've found a segfault in v2.0.1. I believe the issue is a no-ssl directive 
>> on a server line after seeing check ssl on default-server in defaults. 
>> Here's the snips of my config. I haven't been able to create a minimal 
>> config that recreates it, since my config is rather complex.
>> 
>> defaults
>>  log  global
>>  mode http
>>  default-server ca-file ca-certificates.crt resolvers default inter 5s 
>> fastinter 2s downinter 10s init-addr libc,last check ssl check-alpn http/1.1 
>> pool-purge-delay 60s max-reuse 1500 alpn http/1.1
>> […snip…]
>> backend varnish
>>  server varnish_local   unix@/path-to-socket.sock no-check-ssl no-ssl
>> 
>> If I remove no-ssl, it starts up, but the check naturally fails. If I add it 
>> back, I get a segmentation fault. I've tried this with and without unix 
>> sockets to verify it wasn't something related to IP binding.
>> 
>> I'm happy to try alternatives / test things a bit.
>> 
>> Best,
> 
> Indeed, "check-alpn" failed to make sure we were really using a SSL connection
> before attempting to change the ALPN. This should be fixed by commit
> c50eb73b85f80ac1ac6e519fcab2ba6807f5de65, and should be backported to 2.0
> soon.
> 
> Thanks a lot !
> 
> Olivier

Re: Config Segmentation Fault [2.0.1]

[Olivier Houchard]

Hi Luke,

On Fri, Jun 28, 2019 at 07:05:32AM +0200, Luke Seelenbinder wrote:
> Hello all,
> 
> I've found a segfault in v2.0.1. I believe the issue is a no-ssl directive on 
> a server line after seeing check ssl on default-server in defaults. Here's 
> the snips of my config. I haven't been able to create a minimal config that 
> recreates it, since my config is rather complex.
> 
> defaults
>   log  global
>   mode http
>   default-server ca-file ca-certificates.crt resolvers default inter 5s 
> fastinter 2s downinter 10s init-addr libc,last check ssl check-alpn http/1.1 
> pool-purge-delay 60s max-reuse 1500 alpn http/1.1
> […snip…]
> backend varnish
>   server varnish_local   unix@/path-to-socket.sock no-check-ssl no-ssl
> 
> If I remove no-ssl, it starts up, but the check naturally fails. If I add it 
> back, I get a segmentation fault. I've tried this with and without unix 
> sockets to verify it wasn't something related to IP binding.
> 
> I'm happy to try alternatives / test things a bit.
> 
> Best,

Indeed, "check-alpn" failed to make sure we were really using a SSL connection
before attempting to change the ALPN. This should be fixed by commit
c50eb73b85f80ac1ac6e519fcab2ba6807f5de65, and should be backported to 2.0
soon.

Thanks a lot !

Olivier

Requisition for Steps to Install HAProxy in AIX

[Sriram S]

Hi Team,

We're using OS - AIX 7, Model - Power 7.

We wish to install HAProxy 2.0.1 (
http://www.haproxy.org/download/2.0/src/haproxy-2.0.1.tar.gz) in our AIX
box.

This is the first time we're going to install the HAProxy and wish to get
any steps with which we can pursue on with our installation of HAProxy in
AIX box.

Please do the needful at the earliest.

-- 
Regards,
Sriraam S

Re: [PATCH]: hathread little debug feature proposal

[David CARLIER]

do not know if this is the trend here but ... ping :-)

On Fri, 14 Jun 2019 at 12:07, Willy Tarreau  wrote:
>
> Hi David,
>
> On Fri, Jun 14, 2019 at 10:04:41AM +, David CARLIER wrote:
> > Hi,
> >
> > Here a little patch proposal to give names to threads for debug builds.
> > If accepted it is not crucial can be seen after the soon 2.0 release.
>
> I didn't know about this feature. Given that it's only for debugging,
> I suggest we take it later. I'd rather avoid the usual last minute
> change that manages to unexpectedly break monitoring scripts or anything
> but I agree it can be quite useful during debugging.
>
> Thanks,
> Willy