Re: 1.5.9 crashes every 4 hours, like clockwork

ailing Lists mailto:haproxy@formilux.org>> Subject: Re: 1.5.9 crashes every 4 hours, like clockwork Did some digging, and I did find this article: http://blog.tinola.com/?e=36 It could be related to the issue you are experiencing, especially since just before the SIGABRT, the process is tryin

Re: 1.5.9 crashes every 4 hours, like clockwork

:t...@clarridge.ca>> Cc: HAproxy Mailing Lists mailto:haproxy@formilux.org>> Subject: Re: 1.5.9 crashes every 4 hours, like clockwork I ran strace on it just before CRASHTIME. It stopped on cue, with an exit code of 134. The strace output is here: haproxy strace - Pastebin.com<

Re: 1.5.9 crashes every 4 hours, like clockwork

Yes we're on Centos 6.4. Removing the chroot option in the haproxy.cfg appears to have cured it - it didn't crash at 13:00 as it usually does. Many thanks for the help. On Friday, 12 December 2014, 12:00, Lukas Tribus wrote: David, are you on Centos 6.4? Try setting your ciphers t

RE: 1.5.9 crashes every 4 hours, like clockwork

David, are you on Centos 6.4? Try setting your ciphers to one of the Mozilla recommendation [1], that way, you won't negiotating kerberos ciphers suites anymore. If that doesn't help, and you need chroot, then you will have to upgrade to at least Centos 6.5. Regards, Lukas [1] https://wiki

Re: 1.5.9 crashes every 4 hours, like clockwork

Will try without chroot. Contents of /etc/nsswitch.conf: passwd:     filesshadow:     filesgroup:      files hosts:      files dns bootparams: nisplus [NOTFOUND=return] files ethers:     filesnetmasks:   filesnetworks:   filesprotocols:  filesrpc:         filesservices:   files netgroup:   nisplus

Re: 1.5.9 crashes every 4 hours, like clockwork

On 12/12/2014 12:06 PM, Lukas Tribus wrote: Hi David, I've just take a look in openssl code and your trace. The trace shows that process try to open 'krb5.conf' and in openssl/ssl/kssl.c there is some 'abort()' calls. Do you use openssl kerberos5 auth feature of openssl? Wait a minute, that s

RE: 1.5.9 crashes every 4 hours, like clockwork

> Hi David, > > I've just take a look in openssl code and your trace. The trace shows > that process try to open 'krb5.conf' and in openssl/ssl/kssl.c there is > some 'abort()' calls. > > Do you use openssl kerberos5 auth feature of openssl? Wait a minute, that sounds familiar ... http://marc.inf

Re: 1.5.9 crashes every 4 hours, like clockwork

On 12/12/2014 11:08 AM, Emeric Brun wrote: On 12/12/2014 07:16 AM, Vincent Bernat wrote: ❦ 12 décembre 2014 02:08 GMT, David Adams : I ran strace on it just before CRASHTIME. It stopped on cue, with an exit code of 134. The strace output is here: haproxy strace - Pastebin.com As you'll se

Re: 1.5.9 crashes every 4 hours, like clockwork

On 12/12/2014 07:16 AM, Vincent Bernat wrote: ❦ 12 décembre 2014 02:08 GMT, David Adams : I ran strace on it just before CRASHTIME. It stopped on cue, with an exit code of 134. The strace output is here: haproxy strace - Pastebin.com As you'll see, it looks very strange - immediately after

Re: 1.5.9 crashes every 4 hours, like clockwork

❦ 12 décembre 2014 02:08 GMT, David Adams  : > I ran strace on it just before CRASHTIME. It stopped on cue, with an > exit code of 134. > > The strace output is here: haproxy strace - Pastebin.com > > As you'll see, it looks very strange - immediately after a series of > futex calls (I've no idea

Re: 1.5.9 crashes every 4 hours, like clockwork

I ran strace on it just before CRASHTIME.  It stopped on cue, with an exit code of 134. The strace output is here:  haproxy strace - Pastebin.com As you'll see, it looks very strange - immediately after a series of futex calls (I've no idea of their significance, only noting that they don't appea

Re: 1.5.9 crashes every 4 hours, like clockwork

I have an easy workarround : rename the version to out of hour range numbers : 25.26.27 ;) On Thu, Dec 11, 2014 at 4:55 PM, Emeric Brun wrote: > On 12/11/2014 04:20 PM, cyril.bo...@free.fr wrote: >> >> Hi David, >> >> Le 11/12/2014 15:26, David Adams a écrit : >>> >>> We are running 1.5.9 on Cent

Re: 1.5.9 crashes every 4 hours, like clockwork

> Am 11.12.2014 um 15:26 schrieb David Adams >: > > We are running 1.5.9 on Centos 6.5. It crashes 10 seconds (give or take a > few seconds) after 1am, 5am, 9am, 1pm, 5pm and 9pm, like clockwork; let's > call that CRASHTIME. Previously we'd been using 1.5.3 on the sam

Re: 1.5.9 crashes every 4 hours, like clockwork

On 11 Dec 2014 14:27, "David Adams" wrote: > > We are running 1.5.9 on Centos 6.5. It crashes 10 seconds (give or take a few seconds) after 1am, 5am, 9am, 1pm, 5pm and 9pm, like clockwork; let's call that CRASHTIME. Previously we'd been using 1.5.3 on the same hardware for some months without cr

RE: 1.5.9 crashes every 4 hours, like clockwork

> I will do next time. And yes, was planning to run strace. > > Do I need to recompile to enable coredumps? No, you just adjust ulimit before you start, and make sure you didn't strip (as in the command strip) the executable. Then check the core with: gdb path/to/the/binary path/to/the/core

Re: 1.5.9 crashes every 4 hours, like clockwork

I will do next time.  And yes, was planning to run strace.   Do I need to recompile to enable coredumps? On Thursday, 11 December 2014, 17:19, Tait Clarridge wrote: On Thu, Dec 11, 2014 at 12:03 PM, David Adams wrote: > I tried this.  I ran it like this: > > /usr/local/sbin/haproxy

Re: 1.5.9 crashes every 4 hours, like clockwork

❦ 11 décembre 2014 17:03 GMT, David Adams  : > I tried this. I ran it like this: > > /usr/local/sbin/haproxy -db -f /etc/haproxy/haproxy.cfg > > which obviously didn't return as the process ran. Then at the > crashtime (a few seconds past 17:00), that process terminated and the > terminal just sh

Re: 1.5.9 crashes every 4 hours, like clockwork

On Thu, Dec 11, 2014 at 12:03 PM, David Adams wrote: > I tried this. I ran it like this: > > /usr/local/sbin/haproxy -db -f /etc/haproxy/haproxy.cfg > > which obviously didn't return as the process ran. Then at the crashtime (a > few seconds past 17:00), that process terminated and the terminal

Re: 1.5.9 crashes every 4 hours, like clockwork

I tried this.  I ran it like this: /usr/local/sbin/haproxy -db -f /etc/haproxy/haproxy.cfg which obviously didn't return as the process ran.  Then at the crashtime (a few seconds past 17:00), that process terminated and the terminal just showed: aborted I then restarted the process in the normal

Re: 1.5.9 crashes every 4 hours, like clockwork

On 12/11/2014 04:20 PM, cyril.bo...@free.fr wrote: Hi David, Le 11/12/2014 15:26, David Adams a écrit : We are running 1.5.9 on Centos 6.5. It crashes 10 seconds (give or take a few seconds) after 1am, 5am, 9am, 1pm, 5pm and 9pm, like clockwork; let's call that CRASHTIME. Previously we'd be

Re: 1.5.9 crashes every 4 hours, like clockwork

David: You could try running strace -faep HAPROXY_PID -o /tmp/haproxy.strace from 1 minute before the process is getting killed. Then examine the output file for clues. I might not have the options 100% correct. -Jerry Jerry Champlin Absolute Performance Inc. Phone: 303-565-4401 -- Enabling

Re: 1.5.9 crashes every 4 hours, like clockwork

On Thu, Dec 11, 2014 at 10:40 AM, David Adams wrote: > Yes, I am quite sure it's something external to the haproxy process which is > somehow interfering with it, but I can't figure out what. As I mentioned, > looking at a list of processes before and after haproxy stops doesn't reveal > any chan

Re: 1.5.9 crashes every 4 hours, like clockwork

On Thu, 11 Dec 2014, 16:34:02 +0100, David Adams wrote: > Nothing in /var/log/messages related to this.  The only unusual thing is it > reporting that haproxy is starting up again after it stopped.   what type of init process are you using (sysvinit vs. systemd)? Please also check any crontabs/-

Re: 1.5.9 crashes every 4 hours, like clockwork

Yes, I am quite sure it's something external to the haproxy process which is somehow interfering with it, but I can't figure out what.  As I mentioned, looking at a list of processes before and after haproxy stops doesn't reveal any changes in what was running, i.e. it's not like some script run

Re: 1.5.9 crashes every 4 hours, like clockwork

> Meaning the first crash was with 1.5.3, correct? Yes.  It started happening with 1.5.3 (which had been stable for a while), and continues to happen with 1.5.9. > Ok, a part from the informations Oliver requested please provide: > - "haproxy -vv" output HA-Proxy version 1.5.9 2014/11/25Copyr

Re: 1.5.9 crashes every 4 hours, like clockwork

The process just disappears.   I've execute those commands and will see if it generates a coredump when it does it again, which will be in 90 minutes time. On Thursday, 11 December 2014, 14:35, Olivier wrote: Hello, 2014-12-11 15:26 GMT+01:00 David Adams : We are running 1.5.9 on Ce

Re: 1.5.9 crashes every 4 hours, like clockwork

On Thu, Dec 11, 2014 at 4:22 PM, wrote: > Le 11/12/2014 16:20, cyril.bo...@free.fr a écrit : >> 1am, 5am, 9am, >> 1pm, 5pm, 9pm >> 1.5.9 >> => coincidence ? This really make me think of a script. > > I mean "a script *issue*" ;-) > > > -- > Cyril Bonté > mhh David may have enabled the gl

Re: 1.5.9 crashes every 4 hours, like clockwork

Nothing in /var/log/messages related to this.  The only unusual thing is it reporting that haproxy is starting up again after it stopped.   On Thursday, 11 December 2014, 14:37, Olivier wrote: 2014-12-11 15:35 GMT+01:00 Olivier : When you say "crash", do you mean Segfault ?  Als

Re: 1.5.9 crashes every 4 hours, like clockwork

Le 11/12/2014 16:20, cyril.bo...@free.fr a écrit : > 1am, 5am, 9am, > 1pm, 5pm, 9pm > 1.5.9 > => coincidence ? This really make me think of a script. I mean "a script *issue*" ;-) -- Cyril Bonté

Re: 1.5.9 crashes every 4 hours, like clockwork

Hi David, Le 11/12/2014 15:26, David Adams a écrit : > We are running 1.5.9 on Centos 6.5. It crashes 10 seconds (give or take a > few seconds) after 1am, 5am, 9am, 1pm, 5pm and 9pm, like clockwork; let's > call that CRASHTIME. Previously we'd been using 1.5.3 on the same hardware > for some

RE: 1.5.9 crashes every 4 hours, like clockwork

> We are running 1.5.9 on Centos 6.5. It crashes 10 seconds (give or > take a few seconds) after 1am, 5am, 9am, 1pm, 5pm and 9pm, like > clockwork; let's call that CRASHTIME. Previously we'd been using 1.5.3 > on the same hardware for some months without crashes. Once the crashes > start

Re: 1.5.9 crashes every 4 hours, like clockwork

2014-12-11 15:35 GMT+01:00 Olivier : > When you say "crash", do you mean Segfault ? > Also check /var/log/messages to see if OOM Killer is not involved in what you are experiencing.

Re: 1.5.9 crashes every 4 hours, like clockwork

Hello, 2014-12-11 15:26 GMT+01:00 David Adams : > We are running 1.5.9 on Centos 6.5. It crashes 10 seconds (give or take a > few seconds) after 1am, 5am, 9am, 1pm, 5pm and 9pm, like clockwork; > When you say "crash", do you mean Segfault ? Have you tried generating a coredump of the process