Hi Donovan, On Fri, Oct 30, 2015 at 06:39:10PM +0000, Donovan Meyers wrote: > > Hey all, > > I'm currently investigating some possible crashing in 1.5.14. > > I really hate to call it a crash since I've never seen haproxy crash in my > life until now, but the process is suddenly not running and I can't figure > out why.
Well, crashes are indeed extremely rare but sometimes a bug surfaces and causes this. Then we fix it and emit a new version. > I've checked syslogs and puppet logs (and now disabled puppet) and anything > else that could stop the process for any reason. There are no messages > related to haproxy stopping at all. Strange, because if it crashes (segfault, bus error, abort) the kernel should emit a line saying the process died, unless kernel.print-fatal-signals is not set. > I turned haproxy logging up to debug but there's nothing in the log when it > happens. I keep a stats page reloading on my desktop, so the log has entries > for those requests, and then suddenly nothing. So that looks like a crash or a kill from another process. > The issue doesn't seem related to load; under load testing the issue doesn't > manifest (not right away anyway) and it does manifest under no load at all. > > The config is extremely simple; I'm happy to post it if anyone's interested. > But it's not doing anything interesting. That can indeed help. Don't forget to anonymize it if required (eg: stats password). > I've now enabled core dumps and even have it running under valgrind memcheck > but unfortunately I haven't gotten it to happen since I did that. OK. Please test by hand that your core dumps work (just send SIG11 to the running process). You generally need to disable chroot, and enable fs.suid_dumpable if you have either uid/user/gid/group in the config. > The binary is one I compiled from source in order to build an RPM, using the > haproxy.spec from the CentOS 6.5 haproxy-1.4.24-2.el6.src.rpm, slightly > tweaked. For valgrind I also added CFLAGS="-g -O0". OK. > I really hope I can figure out that it's my fault. Barring that, I hope I > can ferret out an issue with haproxy and contribute that to the community > that has been so helpful to me. (Although you won't see a patch from me; I'm > no developer! :) ) We hope so as well. A dying process is not something expected at all and we take such issues very seriously. Note, there are some fixes pending since 1.5.14, I should have issued 1.5.15 already but lack of time, usual excuse, etc... The only important fix pending there is related to option http-send-name-header, so if you're not using it, I don't think you're experiencing a known bug. If the bug happens rarely, you may also want to capture traffic using tcpdump in order to be able to isolate the request(s) that causes this issue (if at all). This can be cumbersome though since you'll have to rotate files to avoid filling the file system. Willy
