Re: HAProxy 1.8.4 crashing

Hi Praveen, There are several fixes for segfaults which might occur in your version of HAProxy. Before checking anything else, you should upgrade to the latest version of HAProxy 1.8 (currently 1.8.12). See http://www.haproxy.org/bugs/bugs-1.8.4.html for bugs fixed in this version compared to

Re: HAProxy 1.8.4 crashing

Hi Praveen, On Thu, Jul 05, 2018 at 04:13:25PM +, UPPALAPATI, PRAVEEN wrote: > > > Hi Haproxy Team, > > Our Prod Haproxy instance is crashing with following error in > /var/log/messages: > > Jun 28 17:52:30 zlp32359 kernel: haproxy[55940]: segfault at 60 ip > 0045b0a9 sp

HAProxy 1.8.4 crashing

Hi Haproxy Team, Our Prod Haproxy instance is crashing with following error in /var/log/messages: Jun 28 17:52:30 zlp32359 kernel: haproxy[55940]: segfault at 60 ip 0045b0a9 sp 7f4ef6b9f010 error 4 in haproxy[40+12b000] Jun 28 17:56:01 zlp32359 systemd: Started Session 73792

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Robin, > De: "Robin Geuze" <rob...@transip.nl> > À: "Willy Tarreau" <w...@1wt.eu> > Cc: haproxy@formilux.org > Envoyé: Lundi 9 Avril 2018 10:24:43 > Objet: Re: Haproxy 1.8.4 crashing workers and increased memory usage > > Hey Willy, &

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hey, Won't that be a bit pointless since we don't use threads? Regards, Robin Geuze On 4/9/2018 10:31, Илья Шипицин wrote: can you try thread sanitizer (in real time)? https://github.com/google/sanitizers/wiki#threadsanitizer I'd like to try myself, however, we do not observe bad things

Re: Haproxy 1.8.4 crashing workers and increased memory usage

can you try thread sanitizer (in real time)? https://github.com/google/sanitizers/wiki#threadsanitizer I'd like to try myself, however, we do not observe bad things in our environment 2018-04-09 13:24 GMT+05:00 Robin Geuze : > Hey Willy, > > So I made a build this morning

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hey Willy, So I made a build this morning with libslz and re-enabled compression and within an hour we had the exit code 134 errors, so zlib does not seem to be the problem here. Regards, Robin Geuze On 4/7/2018 00:30, Willy Tarreau wrote: Hi Robin, On Fri, Apr 06, 2018 at 03:52:33PM

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Robin, On Fri, Apr 06, 2018 at 03:52:33PM +0200, Robin Geuze wrote: > Hey Willy, > > I was actually the one that had the hunch to disable compression. I > suspected that this was the issue because there was a bunch of "abort" calls > in include/common/hathreads.h" which is used by the

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hey Willy, I was actually the one that had the hunch to disable compression. I suspected that this was the issue because there was a bunch of "abort" calls in include/common/hathreads.h" which is used by the compression stuff. However I just noticed those aborts are actually only there if

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Frank, On Fri, Apr 06, 2018 at 10:53:36AM +, Frank Schreuder wrote: > We tested haproxy 1.8.6 with compression enabled today, within the first few > hours it already went wrong: > [ALERT] 095/120526 (12989) : Current worker 5241 exited with code 134 OK thanks, and sorry for that. > Our

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Willy, >> There are very few abort() calls in the code : >>   - some in the thread debugging code to detect recursive locks ; >>   - one in the cache applet which triggers on an impossible case very >>     likely resulting from cache corruption (hence a bug) >>   - a

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Frank, On Thu, Apr 05, 2018 at 09:41:25AM +, Frank Schreuder wrote: > Hi Willy > > There are very few abort() calls in the code : >   - some in the thread debugging code to detect recursive locks ; >   - one in the cache applet which triggers on an impossible case very >

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Willy There are very few abort() calls in the code :   - some in the thread debugging code to detect recursive locks ;   - one in the cache applet which triggers on an impossible case very     likely resulting from cache corruption (hence a bug)   - a few inside the

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Op 23/02/2018 om 13:10 schreef Frank Schreuder: > Hi Willy, > A few more things on the core dumps :   - they are ignored if you have a chroot statement in the global section   - you need not to use "user/uid/group/gid" otherwise the system also    disables core dumps >>> I'm

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Frank, On Fri, Feb 23, 2018 at 12:10:13PM +, Frank Schreuder wrote: > > Well, at least you don't use threads nor lua nor caching nor HTTP/2 so > > it cannot come from any of those we have identified. It could still come > > from openssl however. > > There are some bugfixes marked as

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Frank, On Fri, Feb 23, 2018 at 10:28:15AM +, Frank Schreuder wrote: > > A few more things on the core dumps : > >  - they are ignored if you have a chroot statement in the global section > >  - you need not to use "user/uid/group/gid" otherwise the system also > >    disables core dumps >

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Willy and Tim, > > >> Code 134 implies the worker was killed with SIGABRT. You could check > > >> whether there is a core dump. > > > > > > I don't have any core dumps. > > > > Check whether coredumps are enabled using `ulimit -c`, often they are > > disabled by default, because they could

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi guys, On Thu, Feb 22, 2018 at 04:20:07PM +0100, Tim Düsterhus wrote: > Frank, > > Am 22.02.2018 um 15:33 schrieb Frank Schreuder: > >> Code 134 implies the worker was killed with SIGABRT. You could check > >> whether there is a core dump. > > > > I don't have any core dumps. > > Check

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Frank, Am 22.02.2018 um 15:33 schrieb Frank Schreuder: >> Code 134 implies the worker was killed with SIGABRT. You could check >> whether there is a core dump. > > I don't have any core dumps. Check whether coredumps are enabled using `ulimit -c`, often they are disabled by default, because

Re: Haproxy 1.8.4 crashing workers and increased memory usage

Hi Tim, >> I'm running haproxy 1.8.4 with a heavy work load. >> For some reason some workers die every now and then with the following error >> in the log: >> Feb 22 05:00:42 hostname haproxy[9950]: [ALERT] 052/045759 (9950) : Current >> worker 3569 exited with code 134 >> > > Code 134 implies