Re: [BUG] memory leak with treads and stick-table/peers

[Emmanuel Hocdet]

Hi,

no more leak after "BUG/MINOR: memory: Set objects size for pools in the 
per-thread cache" 

++
Manu


> Le 5 juin 2019 à 16:13, Emmanuel Hocdet  a écrit :
> 
> 
>> Le 5 juin 2019 à 16:07, Emmanuel Hocdet > > a écrit :
>> 
>> Hi Frederic
>> 
>>> Le 5 juin 2019 à 15:44, Frederic Lecaille >> > a écrit :
>>> 
>>> On 6/5/19 3:06 PM, Emmanuel Hocdet wrote:
 Hi,
>>> 
>>> Hi Emmanuel,
>>> 
 After switched to haproxy 1.9 with threads activated, i noticed a 
 significant memory leak.
>>> 
>>> Is valgrind able to expose this memory leak?
>>> 
 With threads disable (and bind process omitted) leak disappear.
 This seems to be related to stick-table/peers with regard to the 
 (simplified) configuration.
>>> 
>>> Has this been revealed by a git bisect?
>>> 
>> 
>> Report is from production, i need to reproduce this on dev platform to debug 
>> more.
>> 
> 
> As a reminder, the leak was detected after thread enable, on 1.9.5.
> 

Re: [BUG] memory leak with treads and stick-table/peers

[Emmanuel Hocdet]

> Le 5 juin 2019 à 16:07, Emmanuel Hocdet  a écrit :
> 
> Hi Frederic
> 
>> Le 5 juin 2019 à 15:44, Frederic Lecaille > > a écrit :
>> 
>> On 6/5/19 3:06 PM, Emmanuel Hocdet wrote:
>>> Hi,
>> 
>> Hi Emmanuel,
>> 
>>> After switched to haproxy 1.9 with threads activated, i noticed a 
>>> significant memory leak.
>> 
>> Is valgrind able to expose this memory leak?
>> 
>>> With threads disable (and bind process omitted) leak disappear.
>>> This seems to be related to stick-table/peers with regard to the 
>>> (simplified) configuration.
>> 
>> Has this been revealed by a git bisect?
>> 
> 
> Report is from production, i need to reproduce this on dev platform to debug 
> more.
> 

As a reminder, the leak was detected after thread enable, on 1.9.5.

Re: [BUG] memory leak with treads and stick-table/peers

[Emmanuel Hocdet]

Hi Frederic

> Le 5 juin 2019 à 15:44, Frederic Lecaille  a écrit :
> 
> On 6/5/19 3:06 PM, Emmanuel Hocdet wrote:
>> Hi,
> 
> Hi Emmanuel,
> 
>> After switched to haproxy 1.9 with threads activated, i noticed a 
>> significant memory leak.
> 
> Is valgrind able to expose this memory leak?
> 
>> With threads disable (and bind process omitted) leak disappear.
>> This seems to be related to stick-table/peers with regard to the 
>> (simplified) configuration.
> 
> Has this been revealed by a git bisect?
> 

Report is from production, i need to reproduce this on dev platform to debug 
more.

> I am trying to understand the impact of the peers configuration 
> simplification on your configuration because as far as I see you do not use 
> this feature.
> 
> This feature is used only if you have "table" lines in "peers" sections.
> 
> Furthermore this feature has not been backported to 1.9. This is a 2.0 
> specific feature. Or perhaps I have missed something.
> 
> This feature comes with at least this commit:
> 
>MEDIUM: stick-table: Stop handling stick-tables as proxies.
> 

It’s legacy stick-table/peers configuration:

backend ssl-back
...
stick-table type binary len 32 size 1m expire 5m peers front-peers

++
Manu

Re: [BUG] memory leak with treads and stick-table/peers

[Frederic Lecaille]

On 6/5/19 3:06 PM, Emmanuel Hocdet wrote:


Hi,


Hi Emmanuel,

After switched to haproxy 1.9 with threads activated, i noticed a 
significant memory leak.


Is valgrind able to expose this memory leak?


With threads disable (and bind process omitted) leak disappear.

This seems to be related to stick-table/peers with regard to the 
(simplified) configuration.


Has this been revealed by a git bisect?

I am trying to understand the impact of the peers configuration 
simplification on your configuration because as far as I see you do not 
use this feature.


This feature is used only if you have "table" lines in "peers" sections.

Furthermore this feature has not been backported to 1.9. This is a 2.0 
specific feature. Or perhaps I have missed something.


This feature comes with at least this commit:

MEDIUM: stick-table: Stop handling stick-tables as proxies.


Fred.

[BUG] memory leak with treads and stick-table/peers

[Emmanuel Hocdet]

Hi,

After switched to haproxy 1.9 with threads activated, i noticed a significant 
memory leak.
With threads disable (and bind process omitted) leak disappear.

This seems to be related to stick-table/peers with regard to the (simplified) 
configuration.

++
Manu


ENV:
HA-Proxy version 1.9.8-1 2019/05/15 - https://haproxy.org/
Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat 
-Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1 
USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : BoringSSL 7f4f41fa
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.35 2014-04-04
Running on PCRE version : 8.35 2014-04-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
  h2 : mode=HTXside=FE|BE
  h2 : mode=HTTP   side=FE
: mode=HTXside=FE|BE
: mode=TCP|HTTP   side=FE|BE

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace

minimal configuration to reproduce the leak:
global
user haproxy
group haproxy
daemon

# ssl(odd) http(even)
nbthread 2

stats socket /var/run/haproxy_L6.sock mode 660 level admin expose-fd 
listeners

log /dev/log daemon warning

# hard limit
hard-stop-after 2h
maxconn 262144

defaults
log global
log-tag "haproxy_L6"
option dontlognull
mode tcp
maxconn 10
timeout connect 500ms


timeout client 610s
retries 3
timeout server 610s

peers front-peers

peer L6_1 x.y.z.1:943

peer L6_2 x.y.z.2:943

peer L6_3 x.y.z.3:943


frontend ssl-front

bind :443  process 1/odd
bind :::443 v6only process 1/odd


tcp-request inspect-delay 5s
tcp-request content reject if !{ req_ssl_sni -m found }
default_backend ssl-back

backend ssl-back
balance roundrobin
option ssl-hello-chk

stick-table type binary len 32 size 1m expire 5m peers front-peers

acl clienthello req_ssl_hello_type 1
acl serverhello rep_ssl_hello_type 2

tcp-request inspect-delay 5s
tcp-request content accept if clienthello

tcp-response content accept if serverhello

stick on payload_lv(43,1) if clienthello
stick store-response payload_lv(43,1) if serverhello


server ssl_1 a.b.c.1:463 check send-proxy 

server ssl_2 a.b.c.2:463 check send-proxy 

server ssl_3 a.b.c.3:463 check send-proxy 

frontend http-front
bind :80  process 1/even
bind :::80 v6only process 1/even


default_backend http-back


backend http-back
balance roundrobin


server L7_1 a.b.c.1:480 check send-proxy 

server L7_2 a.b.c.1:480 check send-proxy 

server L7_3 a.b.c.3:480 check send-proxy