Possible Lua Memory Leak

[Philip Seidel]

Possible Lua Memory Leak?

We are running HAProxy version 1.7.2 (also tested 1.7.5) and are loading a
Lua file which makes use of 2 actions.  The first action is called on
http-request and the second on http-response.  In the simple test case we
put together the request action stores some data in a table and calls
txn:set_priv to save the state for the transaction.  The response action
calls txn:get_priv in order to access the state that was set in the request
action.  All variables are set to nil once they are no longer needed but it
seems that no matter what kind of load HAProxy is receiving we leak a
little bit of memory with each request.  Eventually, response times begin
to increase to the point where health checks to HAProxy begin to fail and
the server is unresponsive.  We can take the instance out of rotation and
memory still doesn't get reclaimed despite all connections to the frontend
and backend being closed.  We do see garbage collections happening but it
never cleans up enough to stabilize the instance.  The leak is easily
reproducible using a local test instance and JMeter.  The test uses 10
connections with keep-alive at around 1500 r/s.  It takes only a few
minutes to consume over 1.5G of memory.  Any ideas as to what might be
going on here?  Is there something wrong with how we are attempting to
integrate these LUA actions?  I am happy to provide additional information
if anyone is willing to assist with this.  I have posted the configuration
and other files on pastebin.

Configuration - https://pastebin.com/SBtAEZ9R
Lua - https://pastebin.com/64Anbxm4

Re: HAProxy for Centos 7

[Andrew Smalley]

Hello  Manojkumar

I see you have requested haproxy 1.6 for CentOS 7 as you dont have root
access.

It is possible to configure and build in your own home directory, however
without root access you will not be able to start it as a non privileged
user and use ports below 1024 as this requires root access.

You could of-course spin up a VM on your desktop and compile manually or
using the below RPMSPEC file

https://github.com/ITV/rpm-haproxy

But sadly yet again you will not be able to install or start the service as
root without access.



Regards

Andrew Smalley

Loadbalancer.org Ltd.



On 25 April 2017 at 16:39, Manojkumar Gupta  wrote:

> Hello,
>
>
>
> Please can you share the binary compiled copy of HAProxy for Centos 7,
> v1.6 or above.
>
>
>
> I don’t have root access and when using generic platform there are
> multiple packages that I need as dependency and without root its hard to
> get it compiled from source.
>
>
>
> I appreciate your help in advance. I need DNS resolvers to be configured.
>
>
>
> Regards,
>
> Manoj.
>
>
>
>
> DISCLAIMER: The information transmitted is intended only for the person or
> entity to which it is addressed and may contain confidential and/or
> privileged material. Any review, re-transmission, dissemination or other
> use of, or taking of any action in reliance upon this information by
> persons or entities other than the intended recipient is prohibited except
> as set forth below. If you received this in error, please contact the
> sender and destroy any copies of this document and any attachments. Email
> sent or received by LendingClub Corporation, its subsidiaries and
> affiliates is subject to archival, monitoring and/or review by and/or
> disclosure to someone other than the recipient.
>

Re: Syntax to capture tcp-request content? How to access the captured data later?

[Swapnil Dubey]

Just to make it clear that my answer is to solve the problem similar to
what you mentioned in the link w.r.t mqtt. If you problem is different from
that then it might not work.

On Tue, Apr 25, 2017 at 10:59 PM, Swapnil Dubey 
wrote:

> Hi Christian,
>
> I had the similar requirement and I had to add the following statement in
> order to make it work. I have put comment on the top of each statement.
>
>  *# Acl to make haproxy wait till enough information is received from
> client to intercept and route*
>
>   *  acl client_wait req_len gt 28  *
>
> *# Acl to match the data in the payload i.e. match the 28th  byte
> in the payload *
>
> *# and if it matches hex 32 (ascii char 2) then return success.*
>
> *acl last_msisdn req.payload(28,1) -m bin 32 *
>
>   * # Delay the connection for max 10 sec to  receive  and inspect
> the packet and match one of the ACL*
>
>*tcp-request inspect-delay 3s*
>
> *# When payload is received match for acl and accept the
> connection if it matches*
>
>* tcp-request content accept if client_wait*
>
> option tcplog
>
>  *   # Use the  tcp-gcp if acl rule names **last_msisdn** matches*
>
>* use_backend tcp-gcp if **last_msisdn*
>
>
> *Regards,*
>
> *Swapnil*
>
>
>
> On Tue, Apr 25, 2017 at 10:28 PM, Christian Rohmann <
> christian.rohm...@inovex.de> wrote:
>
>> Hello haproxy users!
>>
>> I am trying to get my head around the feature to capture arbitrary
>> tcp-request content.
>> (I've already found a previous thread - http://haproxy.formilux.narkiv
>> e.com/dwcoXP7y/haproxy-with-mqtt where someone asked a something similar
>> to extract data from MQTT requests)
>>
>> In the documentation https://cbonte.github.io/hapro
>> xy-dconv/1.5/configuration.html#4.2-tcp-request%20content
>> Is says that the action "capture" will allow for samples to be captured.
>> In my case I want to capture a few bytes of the payload
>> like it's described for acl samples at https://cbonte.github.io/hapro
>> xy-dconv/1.5/configuration.html#7.3.5-req.payload .
>>
>>
>> After some trial an error I found that config syntax:
>>  tcp-request content capture req.payload_lv(3,8) len 8
>>
>> will at least parse correctly. But how can I access the captured bytes
>> later to i.e. log them to the access log or to use them for load balancing
>> rules?
>>
>>
>>
>>
>> Thanks for any hints
>> Regards
>>
>>
>> Christian
>>
>>
>>
>

Re: Syntax to capture tcp-request content? How to access the captured data later?

[Swapnil Dubey]

Hi Christian,

I had the similar requirement and I had to add the following statement in
order to make it work. I have put comment on the top of each statement.

 *# Acl to make haproxy wait till enough information is received from
client to intercept and route*

  *  acl client_wait req_len gt 28  *

*# Acl to match the data in the payload i.e. match the 28th  byte
in the payload *

*# and if it matches hex 32 (ascii char 2) then return success.*

*acl last_msisdn req.payload(28,1) -m bin 32 *

  * # Delay the connection for max 10 sec to  receive  and inspect the
packet and match one of the ACL*

   *tcp-request inspect-delay 3s*

*# When payload is received match for acl and accept the connection
if it matches*

   * tcp-request content accept if client_wait*

option tcplog

 *   # Use the  tcp-gcp if acl rule names **last_msisdn** matches*

   * use_backend tcp-gcp if **last_msisdn*


*Regards,*

*Swapnil*



On Tue, Apr 25, 2017 at 10:28 PM, Christian Rohmann <
christian.rohm...@inovex.de> wrote:

> Hello haproxy users!
>
> I am trying to get my head around the feature to capture arbitrary
> tcp-request content.
> (I've already found a previous thread - http://haproxy.formilux.narkiv
> e.com/dwcoXP7y/haproxy-with-mqtt where someone asked a something similar
> to extract data from MQTT requests)
>
> In the documentation https://cbonte.github.io/hapro
> xy-dconv/1.5/configuration.html#4.2-tcp-request%20content
> Is says that the action "capture" will allow for samples to be captured.
> In my case I want to capture a few bytes of the payload
> like it's described for acl samples at https://cbonte.github.io/hapro
> xy-dconv/1.5/configuration.html#7.3.5-req.payload .
>
>
> After some trial an error I found that config syntax:
>  tcp-request content capture req.payload_lv(3,8) len 8
>
> will at least parse correctly. But how can I access the captured bytes
> later to i.e. log them to the access log or to use them for load balancing
> rules?
>
>
>
>
> Thanks for any hints
> Regards
>
>
> Christian
>
>
>

Syntax to capture tcp-request content? How to access the captured data later?

[Christian Rohmann]

Hello haproxy users!

I am trying to get my head around the feature to capture arbitrary 
tcp-request content.
(I've already found a previous thread - 
http://haproxy.formilux.narkive.com/dwcoXP7y/haproxy-with-mqtt where 
someone asked a something similar to extract data from MQTT requests)


In the documentation 
https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-tcp-request%20content
Is says that the action "capture" will allow for samples to be captured. 
In my case I want to capture a few bytes of the payload
like it's described for acl samples at 
https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#7.3.5-req.payload 
.



After some trial an error I found that config syntax:
 tcp-request content capture req.payload_lv(3,8) len 8

will at least parse correctly. But how can I access the captured bytes 
later to i.e. log them to the access log or to use them for load 
balancing rules?





Thanks for any hints
Regards


Christian

Re: OpenSSL engine and async support

[Grant Zhang]

Hi Emeric,

> On Apr 25, 2017, at 04:03, Emeric Brun  wrote:
> 
> Hi Grant,
> 
> On 04/10/2017 05:16 PM, Grant Zhang wrote:
>> 
>>> On Apr 10, 2017, at 07:42, Emeric Brun  wrote:
>>> 
>> 
 * openssl version (1.1.0b-e?)
>>> compiled 1.1.0e
 
 
>>> Could you provide patches rebased on current dev master branch?
>> I am kinda busy with other project but will try to provide rebased patches 
>> this week.
>> 
>> Thanks, 
>> 
>> Grant
>> 
> 
> I've made a POC of a soft async engine. Based on dasync engine it launchs a 
> thread on priv_rsa_enc to spread the load on multiple cores.
> 
> Regarding openssl s_server it is efficient and scale very well depending the 
> number of core (1700 rsa2048/s on one core, 7400 on 4 cores)
> 
> But using haproxy i'm still facing the same issue. There is a poor scale as 
> for qat. If i check a top, i see that haproxy uses 100% of one core
> and some of the time 80% of an other but for a very short period.
Does this occur with single haproxy process(nproc=1) and you see haproxy 
process bounce between cores? Or you see haproxy occupying one core and 
occasionally use 80% of another core?

> 
> In my opinion, there is very few parallelized jobs. I hope a clean rebase 
> will fix the issue.
Sorry for the delay. Attached is the rebased patches, on top of the latest git 
head:
http://git.haproxy.org/?p=haproxy.git;a=commit;h=013a84fe939cf393fbcf8deb9b4504941d382777

Wrt scaling, my use case is mainly for offloading the cpu load during 
handshake, and as such spreading over multiple cores is not desirable. I've 
tried increasing nproc for scale out purpose. nproc(6 to 8) gives me the max 
connection rate until it hits the hw limit of the qat card. The connection rate 
increase (close to) linearly with the number of processes.

> 
> On haproxy conn/s side:
> native ssl: 1200 conn/s
Which cipher do you use for testing? For me I use ECDHE-RSA-AES128-GCM-SHA256 
and I see about 500 conn/s. For some other cipher I see a bit higher number(e.g 
AES128-SHA256 550 conn/s), but haven't seen 1200conn/s. Not sure whether cpu/hw 
makes such a big difference.

> qat+async: 1700 conn/s
I see about 2000 conn/s per core, so not much different than your number.

Thanks,

Grant



0001-ssl-add-basic-support-for-OpenSSL-crypto-engine.patch
Description: Binary data


0002-RFC-add-openssl-async-support.patch
Description: Binary data

HAProxy for Centos 7

[Manojkumar Gupta]

Hello,

Please can you share the binary compiled copy of HAProxy for Centos 7, v1.6 or 
above.

I don’t have root access and when using generic platform there are multiple 
packages that I need as dependency and without root its hard to get it compiled 
from source.

I appreciate your help in advance. I need DNS resolvers to be configured.

Regards,
Manoj.



DISCLAIMER: The information transmitted is intended only for the person or 
entity to which it is addressed and may contain confidential and/or privileged 
material. Any review, re-transmission, dissemination or other use of, or taking 
of any action in reliance upon this information by persons or entities other 
than the intended recipient is prohibited except as set forth below. If you 
received this in error, please contact the sender and destroy any copies of 
this document and any attachments. Email sent or received by LendingClub 
Corporation, its subsidiaries and affiliates is subject to archival, monitoring 
and/or review by and/or disclosure to someone other than the recipient.

Re: OpenSSL engine and async support

[Emeric Brun]

Hi Grant,

On 04/10/2017 05:16 PM, Grant Zhang wrote:
> 
>> On Apr 10, 2017, at 07:42, Emeric Brun  wrote:
>>
> 
>>> * openssl version (1.1.0b-e?)
>> compiled 1.1.0e
>>>
>>>
>> Could you provide patches rebased on current dev master branch?
> I am kinda busy with other project but will try to provide rebased patches 
> this week.
> 
> Thanks, 
> 
> Grant
> 

I've made a POC of a soft async engine. Based on dasync engine it launchs a 
thread on priv_rsa_enc to spread the load on multiple cores.

Regarding openssl s_server it is efficient and scale very well depending the 
number of core (1700 rsa2048/s on one core, 7400 on 4 cores)

But using haproxy i'm still facing the same issue. There is a poor scale as for 
qat. If i check a top, i see that haproxy uses 100% of one core
and some of the time 80% of an other but for a very short period.

In my opinion, there is very few parallelized jobs. I hope a clean rebase will 
fix the issue.

On haproxy conn/s side:
native ssl: 1200 conn/s
qat+async: 1700 conn/s
dasync modified+async: 1800 conn/s

R,
Emeric

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

[jaseywang]

Awesome! It seems it's really something related with SSL handshakes, We
open 80 port without https and let cdn connect to our 80 port, everything
seems fine.
I'll update this thread laters.

On Tue, Apr 25, 2017 at 5:23 PM, Willy Tarreau  wrote:

> On Tue, Apr 25, 2017 at 05:00:48PM +0800, jaseywang wrote:
> > Here is the data with debug mode off, still the same issue:
> > https://www.dropbox.com/s/4x0cjfv1o2kmwg3/analytics-debug-off.txt?dl=0
> >
> >
> > Flat profile:
> >
> > Each sample counts as 0.01 seconds.
> >  no time accumulated
> >
> >   %   cumulative   self  self total
> >  time   seconds   secondscalls  Ts/call  Ts/call  name
> >   0.00  0.00 0.00   264321 0.00 0.00  hdr_idx_add
> >   0.00  0.00 0.00   189187 0.00 0.00  strlcpy2
> >   0.00  0.00 0.00   155794 0.00 0.00  ultoa_o
> >   0.00  0.00 0.00   144666 0.00 0.00  ltoa_o
> >   0.00  0.00 0.00   122408 0.00 0.00  http_find_header2
> >   0.00  0.00 0.0083596 0.00 0.00  fd_update_cache
> >   0.00  0.00 0.0083336 0.00 0.00  http_sync_req_state
> >   0.00  0.00 0.0078736 0.00 0.00  eb_delete
> >   0.00  0.00 0.0078198 0.00 0.00  eb32_insert
> >   0.00  0.00 0.0078043 0.00 0.00
> conn_update_data_polling
> >   0.00  0.00 0.0067163 0.00 0.00  conn_fd_handler
> >   0.00  0.00 0.0066824 0.00 0.00  vars_init
> >   0.00  0.00 0.0066768 0.00 0.00  utoa_pad
> >   0.00  0.00 0.0061080 0.00 0.00  http_resync_states
> >   0.00  0.00 0.0061080 0.00 0.00  http_sync_res_state
> (...)
>
> For me this shows a perfectly healthy load balancer experiencing a very
> low load. There's even no connection retries, everything looks OK. I'm
> speechless.
>
> Given that no time was reported in any function, it could be possible
> that all the time is spent in SSL handshakes but I'm even having doubts
> on this now.
>
> Do you see some CPU usage on the machine during the test ? Is the CPU
> where haproxy runs saturated ?
>
> The next step will probably require strace to see where the time is wasted.
> You can run it like this :
>
>strace -Tttvs200 -o strace.log -p $(pidof haproxy)
>
> It will reveal the time spent in each syscall and between them. We may find
> that a full millisecond is lost somewhere, helping to narrow the problem
> down.
>
> Willy
>

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

[Willy Tarreau]

On Tue, Apr 25, 2017 at 05:00:48PM +0800, jaseywang wrote:
> Here is the data with debug mode off, still the same issue:
> https://www.dropbox.com/s/4x0cjfv1o2kmwg3/analytics-debug-off.txt?dl=0
> 
> 
> Flat profile:
> 
> Each sample counts as 0.01 seconds.
>  no time accumulated
> 
>   %   cumulative   self  self total
>  time   seconds   secondscalls  Ts/call  Ts/call  name
>   0.00  0.00 0.00   264321 0.00 0.00  hdr_idx_add
>   0.00  0.00 0.00   189187 0.00 0.00  strlcpy2
>   0.00  0.00 0.00   155794 0.00 0.00  ultoa_o
>   0.00  0.00 0.00   144666 0.00 0.00  ltoa_o
>   0.00  0.00 0.00   122408 0.00 0.00  http_find_header2
>   0.00  0.00 0.0083596 0.00 0.00  fd_update_cache
>   0.00  0.00 0.0083336 0.00 0.00  http_sync_req_state
>   0.00  0.00 0.0078736 0.00 0.00  eb_delete
>   0.00  0.00 0.0078198 0.00 0.00  eb32_insert
>   0.00  0.00 0.0078043 0.00 0.00  conn_update_data_polling
>   0.00  0.00 0.0067163 0.00 0.00  conn_fd_handler
>   0.00  0.00 0.0066824 0.00 0.00  vars_init
>   0.00  0.00 0.0066768 0.00 0.00  utoa_pad
>   0.00  0.00 0.0061080 0.00 0.00  http_resync_states
>   0.00  0.00 0.0061080 0.00 0.00  http_sync_res_state
(...)

For me this shows a perfectly healthy load balancer experiencing a very
low load. There's even no connection retries, everything looks OK. I'm
speechless.

Given that no time was reported in any function, it could be possible
that all the time is spent in SSL handshakes but I'm even having doubts
on this now.

Do you see some CPU usage on the machine during the test ? Is the CPU
where haproxy runs saturated ?

The next step will probably require strace to see where the time is wasted.
You can run it like this :

   strace -Tttvs200 -o strace.log -p $(pidof haproxy)

It will reveal the time spent in each syscall and between them. We may find
that a full millisecond is lost somewhere, helping to narrow the problem down.

Willy

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

[Willy Tarreau]

On Tue, Apr 25, 2017 at 03:32:16PM +0800, jaseywang wrote:
> Here is the analytics data, dropbox link:
> https://www.dropbox.com/s/rn92z42ao5l5rdo/analytics.txt?dl=0

Thank you. Well, everything there looks pretty good, with low CPU usage
everywhere and normal number of calls for each function. I'm noting that
you've got some calls to debug_hdr() implying you were running in debuggingr
mode, which spits all the received headers on stderr. Would you happen to
have this sent to a terminal, to a file or any such thing ? If the thing
consuming the debug output is slow, everything will be slow without eating
CPU and that could perfectly explain the behaviour you're facing.

Debugging is enabled either using the "debug" directive in the global section
or with the "-d" argument on the command line. You definitely need to check
this.

Willy

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

[Willy Tarreau]

On Tue, Apr 25, 2017 at 02:26:53PM +0800, jaseywang wrote:
> yes, really permission issue, here is the gmon.out file:
> https://www.dropbox.com/s/c5wq7oxdjw0ddj0/gmon.out?dl=0

I can't do anything from it since it's specific to your executable.

> not sure how this happen:
> # gprof  /usr/sbin/haproxy /var/lib/haproxy/gmon.out
> gprof: file `/usr/sbin/haproxy' has no symbols

Are you sure it's the haproxy executable you've just built ? It looks
like it was stripped. Just in doubt, try it from the source directory
where you built it instead.

Willy

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

[jaseywang]

yes, really permission issue, here is the gmon.out file:
https://www.dropbox.com/s/c5wq7oxdjw0ddj0/gmon.out?dl=0

not sure how this happen:
# gprof  /usr/sbin/haproxy /var/lib/haproxy/gmon.out
gprof: file `/usr/sbin/haproxy' has no symbols

# gprof  haproxy /var/lib/haproxy/gmon.out
haproxy: No such file or directory

On Tue, Apr 25, 2017 at 1:11 PM, Willy Tarreau  wrote:

> On Tue, Apr 25, 2017 at 12:43:22PM +0800, jaseywang wrote:
> > Build options :
> >   TARGET  = linux2628
> >   CPU = generic
> >   CC  = gcc
> >   CFLAGS  = -pg -O2 -fwrapv -g -fno-strict-aliasing
> > -Wdeclaration-after-statement -DTCP_USER_TIMEOUT=18
> >   OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1
>
> OK.
>
> > When I execute pkill -USR1 haproxy, no gmon.out file created in the file
> > system, Google says need -pg paramater, but from above I have add that,
> not
> > sure why I'm not familiar with gcc and related.
>
> The flag is properly there. It's possible that you've started it from a
> directory where the user declared in the global section has no write
> permissions. Or it's possible that you have a "chroot" directive in your
> global section, that you'll need to temporarily disable. You can
> temporarily
> disable all of "chroot", "user", "uid" to be sure.
>
> Willy
>