Re: 1.5-dev19 crashes
Hi Nick, On Sun, Sep 01, 2013 at 10:47:12PM +0200, Nick Jennings wrote: > Hi all, > > I ended up switching service providers, as I'd had continued memory issues > and problems with the stability of the server I was using. Wow, it's never fun when it happens, especially in virtualized environments where you never know if you're always on the same hardware! > I'm not on a new > VPS, from a different provider, running a different OS (Ubuntu 12.04 LTS). > I'm still compiling from the latest in the git repository, and will report > back if I experience any crashes at all. So far so good. OK. > Sorry I couldn't provide more useful debugging information from my > previous server, but I didn't experience any crash after I'd recompiled > with the debugging flags and with the target as linux26 (though it had only > been running a day or so before the switch, so not enough time to say with > any certainty the problem had been resolved, it probably hadn't). > > Now I'm running a 3.10.x kernel and built haproxy with target=linux2628 > ... and still with the debug flags. So, if I get anther crash I will run > haproxy inside gdb until it crashes again. Perfect, thanks for keeping us informed! Willy
Re: 1.5-dev19 crashes
Hi all, I ended up switching service providers, as I'd had continued memory issues and problems with the stability of the server I was using. I'm not on a new VPS, from a different provider, running a different OS (Ubuntu 12.04 LTS). I'm still compiling from the latest in the git repository, and will report back if I experience any crashes at all. So far so good. Sorry I couldn't provide more useful debugging information from my previous server, but I didn't experience any crash after I'd recompiled with the debugging flags and with the target as linux26 (though it had only been running a day or so before the switch, so not enough time to say with any certainty the problem had been resolved, it probably hadn't). Now I'm running a 3.10.x kernel and built haproxy with target=linux2628 ... and still with the debug flags. So, if I get anther crash I will run haproxy inside gdb until it crashes again. Cheers Nick On Sat, Aug 31, 2013 at 8:36 AM, Willy Tarreau wrote: > Hi, > > On Thu, Aug 29, 2013 at 01:35:49PM +0200, Nick Jennings wrote: > > On Thu, Aug 29, 2013 at 4:40 AM, Amyas wrote: > > > > > Nick Jennings writes: > > > > > > > I'm running CentOS 6 with a 2.6.18 kernel, aside from a few > > > > additional packages via the EPEL, there > > > > are no significant modifications. > > > > # uname -aLinux 2.6.18-308.8.2.el5.028stab101.1 #1 SMP > > > > > > That might be one problem, you have TARGET=linux2628 > > > but are using an older kernel than 2.6.28, it should be > > > > > > And on a classic Linux with SSL and ZLIB support (eg: Red Hat 5.x) : > > > $ make TARGET=linux26 CPU=native USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 > > > > > > > > Thanks for pointing that out, I didn't even notice the difference. I've > > recompiled and running the newly built HAproxy/ > > This is true, though this must not cause a crash. The difference between > linux26 and linux2628 are : > USE_LINUX_SPLICE= implicit > USE_LINUX_TPROXY= implicit > USE_ACCEPT4 = implicit > USE_CPU_AFFINITY= implicit > ASSUME_SPLICE_WORKS= implicit > > So in short nothing critical. All of these features have a fallback to > their older equivalent (eg: splice disables itself if it fails, accept4() > falls back to accept(), etc...). So I'm quite certain you're hitting a > real bug. > > If you can't manage to get a core, for whatever reason, you can run with > gdb instead : > ># gdb --args ./haproxy -db -f haproxy.cfg ... >> run > >crash >> generate-core-file >> quit > > It only requires that you have a window with this. I suspect you can do > this as well using a gdb script though I have never tried. > > Regards, > Willy > >
Re: 1.5-dev19 crashes
Hi, On Thu, Aug 29, 2013 at 01:35:49PM +0200, Nick Jennings wrote: > On Thu, Aug 29, 2013 at 4:40 AM, Amyas wrote: > > > Nick Jennings writes: > > > > > I'm running CentOS 6 with a 2.6.18 kernel, aside from a few > > > additional packages via the EPEL, there > > > are no significant modifications. > > > # uname -aLinux 2.6.18-308.8.2.el5.028stab101.1 #1 SMP > > > > That might be one problem, you have TARGET=linux2628 > > but are using an older kernel than 2.6.28, it should be > > > > And on a classic Linux with SSL and ZLIB support (eg: Red Hat 5.x) : > > $ make TARGET=linux26 CPU=native USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 > > > > > Thanks for pointing that out, I didn't even notice the difference. I've > recompiled and running the newly built HAproxy/ This is true, though this must not cause a crash. The difference between linux26 and linux2628 are : USE_LINUX_SPLICE= implicit USE_LINUX_TPROXY= implicit USE_ACCEPT4 = implicit USE_CPU_AFFINITY= implicit ASSUME_SPLICE_WORKS= implicit So in short nothing critical. All of these features have a fallback to their older equivalent (eg: splice disables itself if it fails, accept4() falls back to accept(), etc...). So I'm quite certain you're hitting a real bug. If you can't manage to get a core, for whatever reason, you can run with gdb instead : # gdb --args ./haproxy -db -f haproxy.cfg ... > run crash > generate-core-file > quit It only requires that you have a window with this. I suspect you can do this as well using a gdb script though I have never tried. Regards, Willy
RE: 1.5-dev19 crashes
Hi Nick, > I'm running CentOS 6 with a 2.6.18 kernel > Linux 2.6.18-308.8.2.el5.028stab101.1 #1 SMP Sun Jun 24 20:25:35 MSD 2012 > x86_64 x86_64 x86_64 GNU/Linux CentOS 6.0 should have a 2.6.32 kernel, are you sure this is CentOS 6? Is this a CentOS 5 installation that has been upgraded to CentOS 6, but the kernel upgrade was skipped? The kernel compile date dates back to June 2012, thats more than a year; I'm sure you are missing some important security fixes. I suggest you update your OS. > # echo "/tmp/core-%e-%s-%u-%g-%p-%t"> /proc/sys/kernel/core_pattern > bash: echo: write error: Operation not permitted > (running as root) Very strange. Any dmesg logs when doing this? Read about core dumping in "man 5 core". If your system is somehow specific, I would hope its documented there. Regards, Lukas
Re: 1.5-dev19 crashes
On Thu, Aug 29, 2013 at 4:40 AM, Amyas wrote: > Nick Jennings writes: > > > I'm running CentOS 6 with a 2.6.18 kernel, aside from a few > > additional packages via the EPEL, there > > are no significant modifications. > > # uname -aLinux 2.6.18-308.8.2.el5.028stab101.1 #1 SMP > > That might be one problem, you have TARGET=linux2628 > but are using an older kernel than 2.6.28, it should be > > And on a classic Linux with SSL and ZLIB support (eg: Red Hat 5.x) : > $ make TARGET=linux26 CPU=native USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 > > Thanks for pointing that out, I didn't even notice the difference. I've recompiled and running the newly built HAproxy/ > As far as the core values problem with /proc - could it be selinux ? > Unfortunately not, it's disabled: # sestatus SELinux status: disabled
Re: 1.5-dev19 crashes
Nick Jennings writes: > I'm running CentOS 6 with a 2.6.18 kernel, aside from a few > additional packages via the EPEL, there > are no significant modifications. > # uname -aLinux 2.6.18-308.8.2.el5.028stab101.1 #1 SMP That might be one problem, you have TARGET=linux2628 but are using an older kernel than 2.6.28, it should be And on a classic Linux with SSL and ZLIB support (eg: Red Hat 5.x) : $ make TARGET=linux26 CPU=native USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 As far as the core values problem with /proc - could it be selinux ?
Re: 1.5-dev19 crashes
On Thu, Aug 29, 2013 at 2:41 AM, Amyas wrote: > Nick Jennings writes: > > > > > Nor can I set the core dump pattern:# echo "/tmp/core-%e-%s-%u-%g-%p-%t" > > > /proc/sys/kernel/core_pattern > > > > bash: echo: write error: Operation not permitted > > > > (running as root) > > What distro/kernel? > Did you check that you can see the value: > cat /proc/sys/kernel/core_pattern > > I'm running CentOS 6 with a 2.6.18 kernel, aside from a few additional packages via the EPEL, there are no significant modifications. # uname -a Linux 2.6.18-308.8.2.el5.028stab101.1 #1 SMP Sun Jun 24 20:25:35 MSD 2012 x86_64 x86_64 x86_64 GNU/Linux And yes, both /proc/sys/kernel/core_pattern and /proc/sys/fs/suid_dumpable exist. abrt and systemd can "hijack" core saving systemd saves it in a "journal" > and abrt in /var/cache/core or /var/spool/core > > There are other gotchas for cores depending on your env. >
Re: 1.5-dev19 crashes
Nick Jennings writes: > > Nor can I set the core dump pattern:# echo "/tmp/core-%e-%s-%u-%g-%p-%t" > /proc/sys/kernel/core_pattern > > bash: echo: write error: Operation not permitted > > (running as root) What distro/kernel? Did you check that you can see the value: cat /proc/sys/kernel/core_pattern abrt and systemd can "hijack" core saving systemd saves it in a "journal" and abrt in /var/cache/core or /var/spool/core There are other gotchas for cores depending on your env.
Re: 1.5-dev19 crashes
Hi Everyone, apologies for not replying sooner, I had my filtering a bit too overbearing and didn't notice the replies. In answer to your question, no actually I had never noticed any crashes of haproxy before build19. Unfortunately I just experienced a crash using the git checkout, it was the first since posting here. So, if it's useful at all, the debug log I collected using the previous command is available here: https://silverbucket.net/haproxy_debug3.log.gz I've just recompiled haproxy using the following command: # make DEBUG=-DDEBUG_FULL CFLAGS="-g -O0" TARGET=linux2628 CPU=native USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 Adjusted ulimit, but cannot set `suid_dumpable` # echo 2 > /proc/sys/fs/suid_dumpable bash: echo: write error: Operation not permitted Nor can I set the core dump pattern: # echo "/tmp/core-%e-%s-%u-%g-%p-%t" > /proc/sys/kernel/core_pattern bash: echo: write error: Operation not permitted (running as root) Anyway, I'm running the haproxy that's been compiled with the debug flags now... Cheers Nick On Wed, Aug 28, 2013 at 1:17 AM, amyas wrote: > Willy Tarreau 1wt.eu> writes: > > >+-+ > >| | > >| HAPROXY MUST NEVER EVER CRASH | > >| | > >+-+ > > I know you have said that in the past, but I don't know if people are used > to > packages with that level of commitment. > Might it be worth adding that to the README file in the "how to report > bugs" > section as you mention on the homepage? > > Also, is it close to time for a new dev release? There is more than a page > or so > of improvements since the last one. > > Thanks as always. > > >
Re: 1.5-dev19 crashes
Willy Tarreau 1wt.eu> writes: >+-+ >| | >| HAPROXY MUST NEVER EVER CRASH | >| | >+-+ I know you have said that in the past, but I don't know if people are used to packages with that level of commitment. Might it be worth adding that to the README file in the "how to report bugs" section as you mention on the homepage? Also, is it close to time for a new dev release? There is more than a page or so of improvements since the last one. Thanks as always.
Re: 1.5-dev19 crashes
Hi, On Mon, Aug 19, 2013 at 06:56:09PM +0200, Lukas Tribus wrote: > Hi! > > > > I've been running 1.5-dev releases for about 6 months now, and have > > been running dev19 since it was released in June. It seems to be > > crashing on me more than the previous builds. > > Are you implying older releases crashed as well, but no so often? That's what scares me as well! Guys, please let me state it again : +-+ | | | HAPROXY MUST NEVER EVER CRASH | | | +-+ So please report here any crash you observe, even if it's on a development version, with configuration or any useful information that may help correlate it with other possible reports, because development once becomes stable and we don't want to see any single crash there. People here take bugs very seriously and will make many efforts to track down and fix a bug causing a crash. So please harrass us with crash reports, we're responsible for these bugs and we must fix them! Thanks, Willy
RE: 1.5-dev19 crashes
Hi! > I've been running 1.5-dev releases for about 6 months now, and have > been running dev19 since it was released in June. It seems to be > crashing on me more than the previous builds. Are you implying older releases crashed as well, but no so often? When updating your build, as per Baptiste suggestion, please configure it so that haproxy dumps a core. How can read here howto do it: http://www.mail-archive.com/haproxy@formilux.org/msg09834.html You don't need to enable debug mode (remove the DEBUG statement from the make line and don't start haproxy in debug mode), as the core dump will provide enough informations anyway (the CFLAGS are important when compiling). You may use -O1 instead of -O0, if you have performance problems. When haproxy crashes again, it should produce a core, which you need to backtrace with gdb or provide core + executable. Cheers, Lukas
Re: 1.5-dev19 crashes
Please report if you have no crashes as well. Baptiste On Mon, Aug 19, 2013 at 4:11 PM, Nick Jennings wrote: > Yeah, I was using the tar.gz, will update from git and rebuild and report > any further crashes. > > Thanks > Nick > > > > On Mon, Aug 19, 2013 at 3:05 PM, Baptiste wrote: >> >> Hi Nick, >> >> Can you confirm whether you installed dev19 from the raw .tar.gz >> archive or the latest one from git? >> Usually, the second option is prefered, so if you chose the first one, >> could you please update it using git then let us know if this version >> still crashes. >> >> Baptiste >> >> >> On Mon, Aug 19, 2013 at 2:31 PM, Nick Jennings >> wrote: >> > Hello, >> > >> > I've been running 1.5-dev releases for about 6 months now, and have >> > been >> > running dev19 since it was released in June. It seems to be crashing on >> > me >> > more than the previous builds. >> > >> > I ran it in debug mode and logged to a file, so hopefully this can help >> > to >> > track down the issue. >> > >> > Here's the log file: >> > https://silverbucket.net/haproxy_debug2.log.gz >> > >> > Here's how I've been running haproxy (to get debug info) >> > # haproxy -d -V -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid > >> > /var/log/haproxy_debug/log 2>&1 >> > >> > I'm using SSL offloading, and websocket proxying, as well as standard >> > http >> > connections passing off to nginx. >> > >> > Let me know if there's anything else I can do to provide more >> > information, >> > though since I don't know how to force the crash, it can sometimes take >> > weeks for it to happen, however it's crashed twice in the past 2 days. >> > >> > Cheers >> > Nick >> > > >
Re: 1.5-dev19 crashes
Yeah, I was using the tar.gz, will update from git and rebuild and report any further crashes. Thanks Nick On Mon, Aug 19, 2013 at 3:05 PM, Baptiste wrote: > Hi Nick, > > Can you confirm whether you installed dev19 from the raw .tar.gz > archive or the latest one from git? > Usually, the second option is prefered, so if you chose the first one, > could you please update it using git then let us know if this version > still crashes. > > Baptiste > > > On Mon, Aug 19, 2013 at 2:31 PM, Nick Jennings > wrote: > > Hello, > > > > I've been running 1.5-dev releases for about 6 months now, and have been > > running dev19 since it was released in June. It seems to be crashing on > me > > more than the previous builds. > > > > I ran it in debug mode and logged to a file, so hopefully this can help > to > > track down the issue. > > > > Here's the log file: > > https://silverbucket.net/haproxy_debug2.log.gz > > > > Here's how I've been running haproxy (to get debug info) > > # haproxy -d -V -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid > > > /var/log/haproxy_debug/log 2>&1 > > > > I'm using SSL offloading, and websocket proxying, as well as standard > http > > connections passing off to nginx. > > > > Let me know if there's anything else I can do to provide more > information, > > though since I don't know how to force the crash, it can sometimes take > > weeks for it to happen, however it's crashed twice in the past 2 days. > > > > Cheers > > Nick > > >
Re: 1.5-dev19 crashes
Hi Nick, Can you confirm whether you installed dev19 from the raw .tar.gz archive or the latest one from git? Usually, the second option is prefered, so if you chose the first one, could you please update it using git then let us know if this version still crashes. Baptiste On Mon, Aug 19, 2013 at 2:31 PM, Nick Jennings wrote: > Hello, > > I've been running 1.5-dev releases for about 6 months now, and have been > running dev19 since it was released in June. It seems to be crashing on me > more than the previous builds. > > I ran it in debug mode and logged to a file, so hopefully this can help to > track down the issue. > > Here's the log file: > https://silverbucket.net/haproxy_debug2.log.gz > > Here's how I've been running haproxy (to get debug info) > # haproxy -d -V -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid > > /var/log/haproxy_debug/log 2>&1 > > I'm using SSL offloading, and websocket proxying, as well as standard http > connections passing off to nginx. > > Let me know if there's anything else I can do to provide more information, > though since I don't know how to force the crash, it can sometimes take > weeks for it to happen, however it's crashed twice in the past 2 days. > > Cheers > Nick >
