Re: High CPU Usage followed by segfault error

[Soji Antony]

FYI, the initial version which we were using before upgrading to 1.8.14
was 1.8.13.
By mistake updated it as 1.8.3 in my first email.

Thanks

On Mon, Oct 15, 2018 at 11:10 PM Soji Antony  wrote:

> Hi Olivier,
>
> Many thanks for your reply.
> Please find the gdb output given below.
>
> # gdb /usr/sbin/haproxy core.dump3.13871
> GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
> .
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/sbin/haproxy...(no debugging symbols
> found)...done.
> [New LWP 13872]
> [New LWP 13873]
> [New LWP 13888]
> [New LWP 13889]
> [New LWP 13890]
> [New LWP 13892]
> [New LWP 13893]
> [New LWP 13894]
> [New LWP 13895]
> [New LWP 13871]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/sbin/haproxy'.
> #0  0x55b79086de43 in _start ()
> (gdb) thread apply all bt
>
> Thread 10 (Thread 0x7f88ee327980 (LWP 13871)):
> #0  0x55b79086dee1 in _start ()
>
> Thread 9 (Thread 0x7f88d37fe700 (LWP 13895)):
> #0  0x55b79086de46 in _start ()
>
> Thread 8 (Thread 0x7f88cb7fe700 (LWP 13894)):
> #0  0x55b79086de46 in _start ()
>
> Thread 7 (Thread 0x7f88d3fff700 (LWP 13893)):
> #0  0x55b79086de43 in _start ()
>
> Thread 6 (Thread 0x7f88e8c25700 (LWP 13892)):
> #0  0x55b79086de43 in _start ()
>
> Thread 5 (Thread 0x7f88e94c5700 (LWP 13890)):
> #0  0x55b79086de46 in _start ()
>
> Thread 4 (Thread 0x7f88e9cc6700 (LWP 13889)):
> #0  0x55b79086de46 in _start ()
>
> Thread 3 (Thread 0x7f88ea4c7700 (LWP 13888)):
> #0  0x55b79086de43 in _start ()
>
> Thread 2 (Thread 0x7f88eacc8700 (LWP 13873)):
> #0  0x55b79086de43 in _start ()
>
> Thread 1 (Thread 0x7f88eb4c9700 (LWP 13872)):
> #0  0x55b79086de43 in _start ()
> (gdb) info threads
>   Id   Target Id Frame
>   10   Thread 0x7f88ee327980 (LWP 13871) 0x55b79086dee1 in _start ()
>   9Thread 0x7f88d37fe700 (LWP 13895) 0x55b79086de46 in _start ()
>   8Thread 0x7f88cb7fe700 (LWP 13894) 0x55b79086de46 in _start ()
>   7Thread 0x7f88d3fff700 (LWP 13893) 0x55b79086de43 in _start ()
>   6Thread 0x7f88e8c25700 (LWP 13892) 0x55b79086de43 in _start ()
>   5Thread 0x7f88e94c5700 (LWP 13890) 0x55b79086de46 in _start ()
>   4Thread 0x7f88e9cc6700 (LWP 13889) 0x55b79086de46 in _start ()
>   3Thread 0x7f88ea4c7700 (LWP 13888) 0x55b79086de43 in _start ()
>   2Thread 0x7f88eacc8700 (LWP 13873) 0x55b79086de43 in _start ()
> * 1Thread 0x7f88eb4c9700 (LWP 13872) 0x55b79086de43 in _start ()
> (gdb)
>
> On Mon, Oct 15, 2018 at 4:58 PM Olivier Houchard 
> wrote:
>
>> Hi,
>>
>> On Sat, Oct 13, 2018 at 01:34:53PM +0530, Soji Antony wrote:
>> > Really appreciate if some one can help with this issue. This is
>> happening
>> > quite frequently on our servers. I have taken a coredump when this
>> happened
>> > last time using 'gcore' command. However unable to share it as it might
>> > have ssl related information. Is there anyway I can remove confidential
>> > information from this coredump file before sharing it. Unfortunately we
>> are
>> > not able to reproduce this issue in a test environment. Also CPU usage
>> was
>> > 100% for all the cores which haproxy was using for around 4hrs though we
>> > have taken it out of rotation from DNS & was not serving any traffic.
>> For
>> > me it looks like something internal to haproxy is causing an infinite
>> loop
>> > causing heavy cpu usage.
>> >
>> > Thanks
>> >
>>
>> Sorry for the late answer.
>> Without sending us the core, if you could at least give use the gdb output
>> of "thread apply all bt" so that we know where each thread is spinning,
>> that
>> may be very useful, and shouldn't leak any confidential information.
>>
>> Thanks !
>>
>> Olivier
>>
>> > [image: Screen Shot 2018-10-12 at 8.13.02 PM.png]
>> >
>> > On Fri, Oct 12, 2018 at 12:01 PM Soji Antony 
>> wrote:
>> >
>> > > Hi Oliver,
>> > >
>> > > Thanks for the suggestion. We have upgraded haproxy to 1.8.14 but
>> seeing
>> > > the same CPU issue again.
>> > > I have found that the segmentation fault which we were seeing earlier
>> is
>> > > not related to the CPU spike as it is happening at different time.
>> Recently
>> > > we had the same issue with one of our haproxy 

Patch : remove useless test

[Mildis]

Hi,

Very minor patch : remove a useless test that always evaluate to false.



0001-MINOR-same-length-test-is-made-just-before.patch
Description: Binary data


Mildis

Re: Few problems seen in haproxy? (threads, connections).

[Willy Tarreau]

Hi again,

finally I got rid of the FD lock for single-threaded accesses (most of
them), and based on Olivier's suggestion, I implemented a per-thread
wait queue, and cache-aligned some list heads to avoid undesired cache
line sharing. For me all of this combined resulted in a performance
increase of 25% on a 12-threads workload. I'm interested in your test
results, all of this is in the latest master.

If you still see LBPRM a lot, I can send you the experimental patch
to move the element inside the tree without unlinking/relinking it
and we can see if that provides any benefit or not (I'm not convinced).

Cheers,
Willy

Re: High CPU Usage followed by segfault error

[Soji Antony]

Hi Olivier,

Many thanks for your reply.
Please find the gdb output given below.

# gdb /usr/sbin/haproxy core.dump3.13871
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/haproxy...(no debugging symbols
found)...done.
[New LWP 13872]
[New LWP 13873]
[New LWP 13888]
[New LWP 13889]
[New LWP 13890]
[New LWP 13892]
[New LWP 13893]
[New LWP 13894]
[New LWP 13895]
[New LWP 13871]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/haproxy'.
#0  0x55b79086de43 in _start ()
(gdb) thread apply all bt

Thread 10 (Thread 0x7f88ee327980 (LWP 13871)):
#0  0x55b79086dee1 in _start ()

Thread 9 (Thread 0x7f88d37fe700 (LWP 13895)):
#0  0x55b79086de46 in _start ()

Thread 8 (Thread 0x7f88cb7fe700 (LWP 13894)):
#0  0x55b79086de46 in _start ()

Thread 7 (Thread 0x7f88d3fff700 (LWP 13893)):
#0  0x55b79086de43 in _start ()

Thread 6 (Thread 0x7f88e8c25700 (LWP 13892)):
#0  0x55b79086de43 in _start ()

Thread 5 (Thread 0x7f88e94c5700 (LWP 13890)):
#0  0x55b79086de46 in _start ()

Thread 4 (Thread 0x7f88e9cc6700 (LWP 13889)):
#0  0x55b79086de46 in _start ()

Thread 3 (Thread 0x7f88ea4c7700 (LWP 13888)):
#0  0x55b79086de43 in _start ()

Thread 2 (Thread 0x7f88eacc8700 (LWP 13873)):
#0  0x55b79086de43 in _start ()

Thread 1 (Thread 0x7f88eb4c9700 (LWP 13872)):
#0  0x55b79086de43 in _start ()
(gdb) info threads
  Id   Target Id Frame
  10   Thread 0x7f88ee327980 (LWP 13871) 0x55b79086dee1 in _start ()
  9Thread 0x7f88d37fe700 (LWP 13895) 0x55b79086de46 in _start ()
  8Thread 0x7f88cb7fe700 (LWP 13894) 0x55b79086de46 in _start ()
  7Thread 0x7f88d3fff700 (LWP 13893) 0x55b79086de43 in _start ()
  6Thread 0x7f88e8c25700 (LWP 13892) 0x55b79086de43 in _start ()
  5Thread 0x7f88e94c5700 (LWP 13890) 0x55b79086de46 in _start ()
  4Thread 0x7f88e9cc6700 (LWP 13889) 0x55b79086de46 in _start ()
  3Thread 0x7f88ea4c7700 (LWP 13888) 0x55b79086de43 in _start ()
  2Thread 0x7f88eacc8700 (LWP 13873) 0x55b79086de43 in _start ()
* 1Thread 0x7f88eb4c9700 (LWP 13872) 0x55b79086de43 in _start ()
(gdb)

On Mon, Oct 15, 2018 at 4:58 PM Olivier Houchard 
wrote:

> Hi,
>
> On Sat, Oct 13, 2018 at 01:34:53PM +0530, Soji Antony wrote:
> > Really appreciate if some one can help with this issue. This is happening
> > quite frequently on our servers. I have taken a coredump when this
> happened
> > last time using 'gcore' command. However unable to share it as it might
> > have ssl related information. Is there anyway I can remove confidential
> > information from this coredump file before sharing it. Unfortunately we
> are
> > not able to reproduce this issue in a test environment. Also CPU usage
> was
> > 100% for all the cores which haproxy was using for around 4hrs though we
> > have taken it out of rotation from DNS & was not serving any traffic. For
> > me it looks like something internal to haproxy is causing an infinite
> loop
> > causing heavy cpu usage.
> >
> > Thanks
> >
>
> Sorry for the late answer.
> Without sending us the core, if you could at least give use the gdb output
> of "thread apply all bt" so that we know where each thread is spinning,
> that
> may be very useful, and shouldn't leak any confidential information.
>
> Thanks !
>
> Olivier
>
> > [image: Screen Shot 2018-10-12 at 8.13.02 PM.png]
> >
> > On Fri, Oct 12, 2018 at 12:01 PM Soji Antony 
> wrote:
> >
> > > Hi Oliver,
> > >
> > > Thanks for the suggestion. We have upgraded haproxy to 1.8.14 but
> seeing
> > > the same CPU issue again.
> > > I have found that the segmentation fault which we were seeing earlier
> is
> > > not related to the CPU spike as it is happening at different time.
> Recently
> > > we had the same issue with one of our haproxy servers and found the
> > > following in strace o/p:
> > >
> > > # haproxy -vv
> > >
> > > HA-Proxy version 1.8.14-1ppa1~trusty 2018/09/23
> > > Copyright 2000-2018 Willy Tarreau 
> > >
> > > Build options :
> > >   TARGET  = linux2628
> > >   CPU = generic
> > >   CC  = gcc
> > >   CFLAGS  = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4
> > > -Wformat -Werror=format-security 

Re: [PATCH] DOC: Fix typo

[Willy Tarreau]

Applied, thank you Bertrand.

Willy

Re: Seamless reload and servers connections status

[Lukas Tribus]

Hello Sébastien,


On Mon, 15 Oct 2018 at 16:40, Sébastien Kurtzemann  wrote:
>> No. Only *restart* closes existing front and backend connections.
>> Reload (both seamless and regular) closes them gracefully, so no
>> request is lost.
>
>
> Okay. I think I confound connections and servers sessions... :(
> Are sessions purged after a seamless reload ?
>
> So is there a way to keep sessions states or to configure haproxy TCP 
> backends to not send a request to a server that is currently handling a 
> connection ?

When haproxy reloads, and your backend server is configured with
maxconn 1 on haproxy:
- the old process will keep 1 connection open to your server, until
all old requests are served (so connections close and the process
exits)
- the new process will open the 1 connection when the first request comes in

So there is a moment when both old and new process are running, where
there are actually 2 connections to your backend server (one from the
old haproxy process, one from the new process).
If you don't want that, you cannot reload, you must restart instead.
This will kill the existing session immediately, only allowing for the
new connection.

There is no way to keep the same TCP session alive across haproxy
reloads or restarts. The old connection must go away and a new
connection must be opened. When you restart, this happens
sequentially, when you reload it's the opposite.


Seamless reload makes it possible for haproxy to pass *LISTENING* (but
not established) sockets from the old to the new process, so that we
don't loose any new incoming connections on the frontend while
reloading. Existing connections and transaction however are still
handled by the old process, until it can close them.



Lukas

Re: Seamless reload and servers connections status

[Sébastien Kurtzemann]

Hi,

On Mon, Oct 15, 2018 at 11:53 AM Lukas Tribus  wrote:

> Hello,
>
>
> On Sat, 13 Oct 2018 at 10:34, Sébastien Kurtzemann  wrote:
> >
> > Hi,
> >
> > I’ve got a question about haproxy "seamless reload" : when this
> > operation is perform does all backend servers connections be reset ?
>
> No. Only *restart* closes existing front and backend connections.
> Reload (both seamless and regular) closes them gracefully, so no
> request is lost.
>

Okay. I think I confound connections and servers sessions... :(
Are sessions purged after a seamless reload ?

So is there a way to keep sessions states or to configure haproxy TCP
backends to not send a request to a server that is currently handling a
connection ?


>
>
> > Our use case:
> > - we use Voyager Ingress for Kubernetes
> > (https://appscode.com/products/voyager/) this last use haproxy to
> > perform balancing
> > - we have configure each servers backend with "maxconn 1" (we only
> > what one connection to be handle by one pod)
> > - when a new pod is schedule, Voyager update backend servers then do a
> > haproxy seamless reload which cause that we lost previous connections
> > states for servers
>
> That's not expected. How exactly does Voyager reload haproxy?
>

Under the hood Voyager launch the following command to reload haproxy:

haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -x /var/run/
haproxy.sock -sf $(cat /var/run/haproxy.pid)



>
> Regards,
> Lukas
>

Regards,
Sébastien

Re: gRPC protocol

[Pavlos Parissis]

On 5/24/18 5:54 PM, Daniel Corbett wrote:
> Hello Aleks,
> 
> 
> On 05/24/2018 10:54 AM, Aleksandar Lazic wrote:
>>
>> I remembert that Willy mentioned this in any of his mail.
>> Do you have any rough timeline, this year, next year something like this
>> ;-)
>>
> 
> We're aiming to have the native internal HTTP representation completed for 
> 1.9 which is slated for
> an -rc1 around the end of September with a potential release between 
> mid-October and the end of
> November.  While I cannot make any promises we're hoping to have gRPC added 
> within this release as
> well.
> 

Can you share with us any update on this? Will HAProxy version 1.9 support gRPC?

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: High CPU Usage followed by segfault error

[Olivier Houchard]

Hi,

On Sat, Oct 13, 2018 at 01:34:53PM +0530, Soji Antony wrote:
> Really appreciate if some one can help with this issue. This is happening
> quite frequently on our servers. I have taken a coredump when this happened
> last time using 'gcore' command. However unable to share it as it might
> have ssl related information. Is there anyway I can remove confidential
> information from this coredump file before sharing it. Unfortunately we are
> not able to reproduce this issue in a test environment. Also CPU usage was
> 100% for all the cores which haproxy was using for around 4hrs though we
> have taken it out of rotation from DNS & was not serving any traffic. For
> me it looks like something internal to haproxy is causing an infinite loop
> causing heavy cpu usage.
> 
> Thanks
> 

Sorry for the late answer.
Without sending us the core, if you could at least give use the gdb output
of "thread apply all bt" so that we know where each thread is spinning, that
may be very useful, and shouldn't leak any confidential information.

Thanks !

Olivier

> [image: Screen Shot 2018-10-12 at 8.13.02 PM.png]
> 
> On Fri, Oct 12, 2018 at 12:01 PM Soji Antony  wrote:
> 
> > Hi Oliver,
> >
> > Thanks for the suggestion. We have upgraded haproxy to 1.8.14 but seeing
> > the same CPU issue again.
> > I have found that the segmentation fault which we were seeing earlier is
> > not related to the CPU spike as it is happening at different time. Recently
> > we had the same issue with one of our haproxy servers and found the
> > following in strace o/p:
> >
> > # haproxy -vv
> >
> > HA-Proxy version 1.8.14-1ppa1~trusty 2018/09/23
> > Copyright 2000-2018 Willy Tarreau 
> >
> > Build options :
> >   TARGET  = linux2628
> >   CPU = generic
> >   CC  = gcc
> >   CFLAGS  = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4
> > -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
> >   OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
> > USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1
> >
> > Default settings :
> >   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> >
> > Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
> > Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
> > OpenSSL library supports TLS extensions : yes
> > OpenSSL library supports SNI : yes
> > Built with Lua version : Lua 5.3.1
> > Built with transparent proxy support using: IP_TRANSPARENT
> > IPV6_TRANSPARENT IP_FREEBIND
> > Encrypted password support via crypt(3): yes
> > Built with multi-threading support.
> > Built with PCRE version : 8.31 2012-07-06
> > Running on PCRE version : 8.31 2012-07-06
> > PCRE library supports JIT : no (libpcre build without JIT?)
> > Built with zlib version : 1.2.8
> > Running on zlib version : 1.2.8
> > Compression algorithms supported : identity("identity"),
> > deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> > Built with network namespace support.
> >
> > Available polling systems :
> >   epoll : pref=300,  test result OK
> >poll : pref=200,  test result OK
> >  select : pref=150,  test result OK
> > Total: 3 (3 usable), will use epoll.
> >
> > Available filters :
> > [SPOE] spoe
> > [COMP] compression
> > [TRACE] trace
> >
> > Strace O/P:
> >
> > [pid 114266] <... sched_yield resumed> ) = 0
> > [pid 114265] sched_yield( 
> > [pid 114267] <... sched_yield resumed> ) = 0
> > [pid 114266] sched_yield( 
> > [pid 114265] <... sched_yield resumed> ) = 0
> > [pid 114267] sched_yield( 
> > [pid 114266] <... sched_yield resumed> ) = 0
> > [pid 114265] sched_yield( 
> > [pid 114267] <... sched_yield resumed> ) = 0
> > [pid 114266] sched_yield( 
> > [pid 114265] <... sched_yield resumed> ) = 0
> > [pid 114267] sched_yield( 
> > [pid 114266] <... sched_yield resumed> ) = 0
> > [pid 114265] sched_yield( 
> > [pid 114267] <... sched_yield resumed> ) = 0
> > [pid 114266] sched_yield( 
> > [pid 114265] <... sched_yield resumed> ) = 0
> > [pid 114267] sched_yield( 
> > [pid 114266] <... sched_yield resumed> ) = 0
> > [pid 114265] sched_yield( 
> > [pid 114267] <... sched_yield resumed> ) = 0
> > [pid 114266] sched_yield( 
> > [pid 114265] <... sched_yield resumed> ) = 0
> > [pid 114267] sched_yield( 
> > [pid 114266] <... sched_yield resumed> ) = 0
> > [pid 114265] sched_yield( 
> > [pid 114267] <... sched_yield resumed> ) = 0
> > [pid 114266] sched_yield( 
> > [pid 114265] <... sched_yield resumed> ) = 0
> > [pid 114267] sched_yield( 
> > [pid 114266] <... sched_yield resumed> ) = 0
> > [pid 114265] sched_yield( 
> > [pid 114267] <... sched_yield resumed> ) = 0
> > [pid 114266] sched_yield( 
> > [pid 114265] <... sched_yield resumed> ) = 0
> > [pid 114267] sched_yield( 
> > [pid 114266] <... sched_yield resumed> ) = 0
> > [pid 114267] <... sched_yield resumed> ) = 0
> > [pid 114266] sched_yield( 
> > [pid 114265] sched_yield( 
> > [pid 114267] sched_yield( 
> > [pid 114266] <... sched_yield resumed> ) = 0
> > [pid 114265] <... sched_yield resumed> ) = 

Re: [ANNOUNCE] haproxy-1.9-dev3

[Willy Tarreau]

Hi Tim,

On Mon, Oct 01, 2018 at 04:44:38PM +0200, Tim Düsterhus wrote:
> in a debian:sid Docker Container:
> 
> > root@df1d20d1da29:/pwd# dpkg -l 'gcc*' |grep '^ii'
> > ii  gcc  4:8.2.0-1amd64GNU C compiler
> > ii  gcc-88.2.0-7  amd64GNU C compiler
> > ii  gcc-8-base:amd64 8.2.0-7  amd64GCC, the GNU Compiler 
> > Collection (base package)
> > root@df1d20d1da29:/pwd# gcc --version
> > gcc (Debian 8.2.0-7) 8.2.0
> > Copyright (C) 2018 Free Software Foundation, Inc.
> > This is free software; see the source for copying conditions.  There is NO
> > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
(...)

A bit late, but thanks for these warnings, I think all of them are
addressed now.

Willy

Re: Seamless reload and servers connections status

[Lukas Tribus]

Hello,


On Sat, 13 Oct 2018 at 10:34, Sébastien Kurtzemann  wrote:
>
> Hi,
>
> I’ve got a question about haproxy "seamless reload" : when this
> operation is perform does all backend servers connections be reset ?

No. Only *restart* closes existing front and backend connections.
Reload (both seamless and regular) closes them gracefully, so no
request is lost.


> Our use case:
> - we use Voyager Ingress for Kubernetes
> (https://appscode.com/products/voyager/) this last use haproxy to
> perform balancing
> - we have configure each servers backend with "maxconn 1" (we only
> what one connection to be handle by one pod)
> - when a new pod is schedule, Voyager update backend servers then do a
> haproxy seamless reload which cause that we lost previous connections
> states for servers

That's not expected. How exactly does Voyager reload haproxy?


Regards,
Lukas