Re: haproxy segfault

Hi Tim, On Fri, Mar 08, 2019 at 01:43:58AM +0100, Tim Düsterhus wrote: > Willy, > > Am 16.02.19 um 06:23 schrieb Willy Tarreau: > > On Tue, Feb 12, 2019 at 10:28:01PM +0100, Lukas Tribus wrote: > >>> Did this bug has been introduced in 1.9.4 ? > >>> I haven't notice this behavior before. > >> >

Re: haproxy segfault

Willy, Am 16.02.19 um 06:23 schrieb Willy Tarreau: > On Tue, Feb 12, 2019 at 10:28:01PM +0100, Lukas Tribus wrote: >>> Did this bug has been introduced in 1.9.4 ? >>> I haven't notice this behavior before. >> >> Yes. Also see the 1.8.19 release notes. > > I forgot we backported it to 1.9.4, I

Re: haproxy segfault

On Tue, Feb 12, 2019 at 10:28:01PM +0100, Lukas Tribus wrote: > > Did this bug has been introduced in 1.9.4 ? > > I haven't notice this behavior before. > > Yes. Also see the 1.8.19 release notes. I forgot we backported it to 1.9.4, I thought it was only queued after 1.9.4, we should have

Re: haproxy segfault

On 19-02-13 16:27:21, Hugues Alary wrote: > (Also, I've been looking for commit 451c5a88 and can't find it > anywhere). See http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=451c5a88, also attached. Cheers, Georg From 451c5a8879a9d59b489ad5117c984044d41c8338 Mon Sep 17 00:00:00 2001 From:

Re: haproxy segfault

Hi Hugues, On Thursday, 14 February 2019, Hugues Alary wrote: > Hi, > > I am also running into this issue using 1.9.4 (i.e. the current "latest" > docker image) with absolutely no load at all (1 client): > > [ALERT] 044/001000 (1) : sendmsg()/writev() failed in logger #1: No such > file or

Re: haproxy segfault

Hi, I am also running into this issue using 1.9.4 (i.e. the current "latest" docker image) with absolutely no load at all (1 client): [ALERT] 044/001000 (1) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2) [NOTICE] 044/001000 (1) : New worker #1 (8) forked [ALERT]

Re: haproxy segfault

Le 13/02/2019 à 09:40, m...@mildis.org a écrit : Thanks Vincent, got a core dump. Here is the backtrace #0  0x5577fb060375 in __pool_get_from_cache (pool=0x5577fb3f7540 ) at include/common/memory.h:199 199    include/common/memory.h: No such file or directory. (gdb) bt full #0 

Re: haproxy segfault

Thanks Vincent, got a core dump. Here is the backtrace #0  0x5577fb060375 in __pool_get_from_cache (pool=0x5577fb3f7540 ) at include/common/memory.h:199 199    include/common/memory.h: No such file or directory. (gdb) bt full #0  0x5577fb060375 in __pool_get_from_cache

Re: haproxy segfault

❦ 12 février 2019 21:44 +01, Mildis : > I'm struggling with Stretch/systemd to generate the coredump on crash. > Even running haproxy by hand with ulimit -c unlimited does not > generate a coredump. Also install haproxy-dbgsym. You need to comment the chroot directive in your HAProxy

Re: haproxy segfault

On Tue, 12 Feb 2019 at 21:46, Mildis wrote: > > > Le 12 févr. 2019 à 21:26, Christopher Faulet a écrit : > > > Hi, > > A recent fix about a double free has been merge in HAProxy 1.9: > > http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=451c5a88 > > Maybe you've hit this bug. > > > Did this

Re: haproxy segfault

> Le 12 févr. 2019 à 21:26, Christopher Faulet a écrit : >> > Hi, > > A recent fix about a double free has been merge in HAProxy 1.9: > > http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=451c5a88 > > > Maybe you've hit

Re: haproxy segfault

> Le 12 févr. 2019 à 18:58, Aleksandar Lazic a écrit : > > Hi. > > Can you activate coredumps > > ulimit -c unlimited > > you should find the core in > > /tmp > > or just search for core on the filesystem > I'm struggling with Stretch/systemd to generate the coredump on crash. Even

Re: haproxy segfault

Le 12/02/2019 à 18:36, Mildis a écrit : Hi list, haproxy is segfaulting multiple times these days for no apparent reason. At first i thought is was a load issue but even few RPS made it crash. Symptoms are always the same : segfault of a worker then spawn of a new. If load is very high,

Re: haproxy segfault

Hi. Am 12.02.2019 um 18:36 schrieb Mildis: > Hi list, > > haproxy is segfaulting multiple times these days for no apparent reason. > At first i thought is was a load issue but even few RPS made it crash. > > Symptoms are always the same : segfault of a worker then spawn of a new. > If load is

haproxy segfault

Hi list, haproxy is segfaulting multiple times these days for no apparent reason. At first i thought is was a load issue but even few RPS made it crash. Symptoms are always the same : segfault of a worker then spawn of a new. If load is very high, spawned worker segfault immediatly. In the

Re: Haproxy segfault error 4 in libc-2.24

Le 04/10/2017 à 05:12, Willy Tarreau a écrit : On Wed, Oct 04, 2017 at 05:07:07AM +0200, Willy Tarreau wrote: On Wed, Oct 04, 2017 at 04:40:53AM +0200, Willy Tarreau wrote: On Tue, Oct 03, 2017 at 06:57:45PM +0200, Marcus Ulbrich wrote: Hey Jarno, This seems to work stable! The idea for this

Re: Haproxy segfault error 4 in libc-2.24

Okay, mh if there is another way to forbid any kind of "sleep" as url parameter not crashing haproxy, I'll use this. Any ideas? Where should I change this? I installed haproxy from repository... Thanks for helping... Marcus

Re: Haproxy segfault error 4 in libc-2.24

On Wed, Oct 04, 2017 at 05:07:07AM +0200, Willy Tarreau wrote: > On Wed, Oct 04, 2017 at 04:40:53AM +0200, Willy Tarreau wrote: > > On Tue, Oct 03, 2017 at 06:57:45PM +0200, Marcus Ulbrich wrote: > > > Hey Jarno, > > > > > > This seems to work stable! > > > The idea for this acl was to prevent

Re: Haproxy segfault error 4 in libc-2.24

On Wed, Oct 04, 2017 at 04:40:53AM +0200, Willy Tarreau wrote: > On Tue, Oct 03, 2017 at 06:57:45PM +0200, Marcus Ulbrich wrote: > > Hey Jarno, > > > > This seems to work stable! > > The idea for this acl was to prevent attackers testing for example MySQL > > injection by sleep command. ,,sleep"

Re: Haproxy segfault error 4 in libc-2.24

On Tue, Oct 03, 2017 at 06:57:45PM +0200, Marcus Ulbrich wrote: > Hey Jarno, > > This seems to work stable! > The idea for this acl was to prevent attackers testing for example MySQL > injection by sleep command. ,,sleep" is in none of our URLs! > Do you have an idea about an acl about this not

Re: Haproxy segfault error 4 in libc-2.24

Hey Jarno, This seems to work stable! The idea for this acl was to prevent attackers testing for example MySQL injection by sleep command. „sleep“ is in none of our URLs! Do you have an idea about an acl about this not crashing haproxy? Thanks a lot! Marcus

Re: Haproxy segfault error 4 in libc-2.24

Hi, On Tue, Oct 03, Marcus Ulbrich wrote: >     #denysleep arguments in url >     acl sleeper url_sub,url_dec,lower -m sub sleep( >     http-request deny if sleeper Stack trace had sample_conv_str2lower: >> #4 sample_conv_str2lower (arg_p=, smp=0x7fff217f4f80, >> private=) at src/sample.c:1518

Re: Haproxy segfault error 4 in libc-2.24

❦ 3 octobre 2017 17:54 +0200, Marcus Ulbrich  : > yes... it crashed after 5mins also without this acl. I was suspecting this ACL as this is the only one with a case-insensitive match. But maybe the same codepath is used when matching header names. > I should test

Re: Haproxy segfault error 4 in libc-2.24

yes... it crashed after 5mins also without this acl. I should test commenting all acl for testing. Is there no way to see what acl was active, when haproxy crashes? Thank you very much so far...

Re: Haproxy segfault error 4 in libc-2.24

❦ 3 octobre 2017 16:34 +0200, Marcus Ulbrich  : >     acl badbots hdr_reg(User-Agent) -i -f /etc/haproxy/badbots.lst >     http-request deny if badbots !whitelistips_agents Try removing this one and check if it still crashes (hoping there is only one crash). --

Re: Haproxy segfault error 4 in libc-2.24

Okay... here is the config: There is another frontend for port 8080... but the same scheme... I left it out here... global     log /dev/log local0     log /dev/log local1 notice     chroot /var/lib/haproxy     stats socket /run/haproxy/admin.sock mode 660 level admin     stats timeout 30s    

Re: Haproxy segfault error 4 in libc-2.24

On Mon, Oct 02, 2017 at 05:16:13PM +0200, Vincent Bernat wrote: > ? 2 octobre 2017 17:06 +0200, Marcus Ulbrich   > : > > > I even get no core dump with the python oneliner either with chroot > > nor without... > > So, kernel.core_pattern seems to be problematic

Re: Haproxy segfault error 4 in libc-2.24

❦ 3 octobre 2017 11:29 +0200, Marcus Ulbrich  : > and here is the coredump with libssl and haproxy... I can not get > clear about this: Not the same one as previously. But this one is entirely in HAProxy. For this one, I think an excerpt of your configuration

Re: Aw: Re: Haproxy segfault error 4 in libc-2.24

❦ 3 octobre 2017 11:15 +0200, lu...@gmx.net : >> Could you get another one with libssl1.1-dbgsym installed? > > Mmmh there is no libssl1.1-dbgsym in stretch, only in sid? > > I do think we need those stack traces from libssl. It should be there. But you need to enable the right repository:

Re: Haproxy segfault error 4 in libc-2.24

Hey, thank you all for helping! okay the output you wish to know: Package: libssl1.1 Source: openssl Version: 1.1.0f-3 Installed-Size: 3524 Maintainer: Debian OpenSSL Team Architecture: amd64 Depends: libc6 (>= 2.14), debconf (>= 0.5) | debconf-2.0

Aw: Re: Haproxy segfault error 4 in libc-2.24

Hello, > Could you get another one with libssl1.1-dbgsym installed? Mmmh there is no libssl1.1-dbgsym in stretch, only in sid? I do think we need those stack traces from libssl. > Okay... I've got a core dump... Thanks a lot!!! > > But what this means? Is the openssl package uptodate? It

Re: Haproxy segfault error 4 in libc-2.24

;m.ulbr...@tu-braunschweig.de> Sent: 2 octobre 2017 19:57 +0200 Subject: Re: Haproxy segfault error 4 in libc-2.24 To: Vincent Bernat Cc: haproxy@formilux.org > Okay... I've got a core dump... Thanks a lot!!! > > But what this means? > > > > Program received sig

Re: Haproxy segfault error 4 in libc-2.24

Okay... I've got a core dump... Thanks a lot!!! But what this means? Program received signal SIGSEGV, Segmentation fault. __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/../multiarch/memmove-vec-unaligned-erms.S:

Re: Haproxy segfault error 4 in libc-2.24

you can try to attach like that https://stackoverflow.com/questions/2152582/start-gdb-using-a-pid (I still think, that core dump is better if possible) 2017-10-02 21:59 GMT+05:00 Marcus Ulbrich : > no chance... error or killing haproxy there is no core file

Re: Haproxy segfault error 4 in libc-2.24

How many haproxy process do you have? Can you reduce nbprocs to 1 if you set it to another value? In this case, attach directly to haproxy with gdb -p pid, type continue to unblock it and wait for the segfault. Then bt full. On October 2, 2017 6:59:47 PM GMT+02:00, Marcus Ulbrich

Re: Haproxy segfault error 4 in libc-2.24

no chance... error or killing haproxy there is no core file there... wether in /var/lib, nor in /tmp I am frustrated...

Re: Haproxy segfault error 4 in libc-2.24

❦ 2 octobre 2017 18:38 +0200, Marcus Ulbrich <m.ulbr...@tu-braunschweig.de> : > yes... core of python script is there than... but i can't get one of > haproxy segfault :-/ Not even in /var/lib/haproxy then? If it works with the Python script, try set kernel.core_pattern to jus

Re: Haproxy segfault error 4 in libc-2.24

yes... core of python script is there than... but i can't get one of haproxy segfault :-/

Re: Haproxy segfault error 4 in libc-2.24

❦ 2 octobre 2017 17:06 +0200, Marcus Ulbrich  : > I even get no core dump with the python oneliner either with chroot > nor without... So, kernel.core_pattern seems to be problematic (unrelated, but my python one-liner wasn't totally correct either). Try again

Re: Haproxy segfault error 4 in libc-2.24

okay, here the outputs: rgrep chroot /etc: /etc/rsyslog.d/49-haproxy.conf:# Create an additional socket in haproxy's chroot in order to allow logging via /etc/rsyslog.d/49-haproxy.conf:# /dev/log to chroot'ed HAProxy processes The machine is only a virtualbox machine... but this was no a

Re: Haproxy segfault error 4 in libc-2.24

e" ――― Original Message ――― From: Marcus Ulbrich <m.ulbr...@tu-braunschweig.de> Sent: 2 octobre 2017 16:39 +0200 Subject: Re: Haproxy segfault error 4 in libc-2.24 To: Vincent Bernat Cc: haproxy@formilux.org > okay... > > $# sysctl kernel.core_pattern > > ke

Re: Haproxy segfault error 4 in libc-2.24

okay... $# sysctl kernel.core_pattern kernel.core_pattern = /tmp/core.%e.%p.%h.%t $# ls -ld /var/lib/haproxy/tmp drwxrwxrwt 2 root root 4096 Okt  2 16:11 /var/lib/haproxy/tmp

Re: Haproxy segfault error 4 in libc-2.24

❦ 2 octobre 2017 16:29 +0200, Marcus Ulbrich  : > sorry, but it is commented out... :( Humm, I don't see how HAProxy would chroot itself without this directive. Let's try to get the core inside the chroot. Could you confirm the output of: sysctl

Re: Haproxy segfault error 4 in libc-2.24

sorry, but it is commented out... :( Am 02.10.2017 um 16:27 schrieb Vincent Bernat: Then, remove the chroot directive from HAProxy configuration.

Re: Haproxy segfault error 4 in libc-2.24

tobre 2017 16:23 +0200 Subject: Re: Haproxy segfault error 4 in libc-2.24 To: Vincent Bernat Cc: haproxy@formilux.org > nope...  /var/lib/haproxy/tmp/ directory is left empty after crash... > > > Am 02.10.2017 um 16:09 schrieb Vincent Bernat: >> ❦ 2 octobre 2017 16:05 +0200

Re: Haproxy segfault error 4 in libc-2.24

nope...  /var/lib/haproxy/tmp/ directory is left empty after crash... Am 02.10.2017 um 16:09 schrieb Vincent Bernat: ❦ 2 octobre 2017 16:05 +0200, Marcus Ulbrich  : this is linked to /proc/20313/root -> /var/lib/haproxy and there is dev/log as empty file..

Re: Haproxy segfault error 4 in libc-2.24

❦ 2 octobre 2017 16:05 +0200, Marcus Ulbrich  : > this is linked to /proc/20313/root -> /var/lib/haproxy > > and there is dev/log as empty file.. Then, create /var/lib/haproxy/tmp: mkdir /var/lib/haproxy/tmp chmod 1777 /var/lib/haproxy/tmp You should get the

Re: Haproxy segfault error 4 in libc-2.24

❦ 2 octobre 2017 15:58 +0200, Marcus Ulbrich  : > Yes there is no core dump... > > i've ched it and ist was both unlimited... And "ls -l /proc/xxx/root" is "/" (where xxx is the PID of one of the HAProxy processes)? -- What good is an obscenity trial except to

Re: Haproxy segfault error 4 in libc-2.24

uger) ――― Original Message ――― From: Marcus Ulbrich <m.ulbr...@tu-braunschweig.de> Sent: 2 octobre 2017 12:49 +0200 Subject: Re: Haproxy segfault error 4 in libc-2.24 To: Vincent Bernat Cc: haproxy@formilux.org > Hello Vincent, > > thanks for your reply. I have

Re: Haproxy segfault error 4 in libc-2.24

Hello Vincent, thanks for your reply. I have done what you said... but there ist nore core file dumped in /tmp/. running gdb console it only says "[Inferior 1 (process 719) exited normally]" when the segfault error occours and haproxy restarts. bt and bt full results in "No stack." Can

Re: Haproxy segfault error 4 in libc-2.24

❦ 2 octobre 2017 10:31 +0200, Marcus Ulbrich  : > I am running haproxy 1.7.9-1~bpo9+1 on debian 9.1. And after running a > while with production data haproxy stops working wiith segmentation > fault: > > haproxy[26291]: segfault at 5562af80e000 ip 7f5985e48149

Haproxy segfault error 4 in libc-2.24

Hello, I am running haproxy 1.7.9-1~bpo9+1 on debian 9.1. And after running a while with production data haproxy stops working wiith segmentation fault: haproxy[26291]: segfault at 5562af80e000 ip 7f5985e48149 sp 7ffe1d613488 error 4 in libc-2.24 Can you please help or have any

RE: HAProxy segfault

Hi David, Hi! HAProxy 1.6-dev1, CentOS6 Getting a segfault when trying connect to port 3389. segfault at 0 ip (null) sp 7fff18a41268 error 14 in haproxy[40+a4000] You are using the development tree, please upgrade latest git first, there are 233 commits since 1.6-dev1.

RE: HAProxy segfault

Thanks. tried this version, it works fine..  Ok, thanks for confirming.

HAProxy segfault

Hi! HAProxy 1.6-dev1, CentOS6 Getting a segfault when trying connect to port 3389. segfault at 0 ip (null) sp 7fff18a41268 error 14 in haproxy[40+a4000] Compiling with next options: make TARGET=linux26 make install the configuration file is as listed below: global daemon

Re: HAProxy segfault

Hello Just an observation You have option tcpka twice Neil On 14 May 2015 14:02, David David dd432...@gmail.com wrote: Hi! HAProxy 1.6-dev1, CentOS6 Getting a segfault when trying connect to port 3389. segfault at 0 ip (null) sp 7fff18a41268 error 14 in haproxy[40+a4000]

haproxy segfault

Hi, we are regularly hitting a bug in haproxy leading to a segfault: [2433667.679923] haproxy[28161]: segfault at 1033eb6d4 ip 004477b0 sp 7fff5584c8e0 error 4 in haproxy[40+a9000] We are running: - haproxy v1.5.3 with multiple frontend/backend, offload SSL ; - on Debian 7:

Re: haproxy segfault

Could you share a frontend/backend configuration template anonymized ? ;) On Fri, Aug 29, 2014 at 5:14 PM, Romain LE DISEZ rom...@ledisez.net wrote: Hi, we are regularly hitting a bug in haproxy leading to a segfault: [2433667.679923] haproxy[28161]: segfault at 1033eb6d4 ip 004477b0

Re: haproxy segfault

Attached is an expunged version of the configuration we use. We have about 20 frontends/backends couple in the real config. Le Vendredi 29 Août 2014 17:32 CEST, Steven Le Roux ste...@le-roux.info a écrit: Could you share a frontend/backend configuration template anonymized ? ;) On Fri, Aug

Re: Re: haproxy segfault

Hi, (gdb) bt #0 http_skip_chunk_crlf (msg=0x388ba50) at src/proto_http.c:2113 #1 http_request_forward_body (s=s@entry=0x388b9a0, req=req@entry=0x343ddf0, an_bit=an_bit@entry=8192) at src/proto_http.c:5398 #2 0x00467298 in process_session (t=0x39eb8d0) at src/session.c:1955 #3

Re: Re: haproxy segfault

It looks the same bug. We cannot have a file larger than 5GB (the backend is an openstack swift installation). But, if the client is using keepalive, I guess he can transfer more than 16GB in the same connection. If we succeed to reproduce the bug that way, we will try the proposed patch.

Re: haproxy segfault issue

Hi Willy, OK got it too now. Interestingly, I never encountered it till now. The same issue exists when sending a POST and a HEAD. This is now fixed with commit cbc743e36. Thanks for reporting this, Willy It works well!! Thanks for your quick patch.. Seri

Re: haproxy segfault issue

Hi, On Fri, Dec 28, 2012 at 12:59:55PM +0900, seri0...@naver.com wrote: Hi!! I've installed haproxy latest snapshot(df97447088ec931320fca6370cc360cb19a82a59) In Stat UI, when I clicked Disable and Apply, segfault happened as below.. kernel: haproxy[9978]: segfault at 2272ff8 ip