Bug#840580: apache2-bin: crashes when issuing a restart while mod_cgid is enabled
On Sunday, 25 November 2018 19:07:56 CET Bernhard Übelacker wrote: > Dear Maintainer, > tried to find out the actual location that the backtrace points to. > > Unfortunately I could not make any clue out of the line > containing /usr/sbin/apache2(+0x29e450). > > But at least, I think, the line containing mod_mpm_prefork.so(+0x4c08) > translates to function prefork_run in server/mpm/prefork/prefork.c. > > As this is a rather big function, and looks like it is never left while > the server runs, and there are no local arrays accessed, the stack > canary may be overwritten by some function called from there. > But the stack canary is just checked when prefork_run exits. > > Kind regards, > Bernhard The "bt full" gdb command is the important one. Please show its output as described in README.backtrace
Bug#840580: apache2-bin: crashes when issuing a restart while mod_cgid is enabled
Dear Maintainer, tried to find out the actual location that the backtrace points to. Unfortunately I could not make any clue out of the line containing /usr/sbin/apache2(+0x29e450). But at least, I think, the line containing mod_mpm_prefork.so(+0x4c08) translates to function prefork_run in server/mpm/prefork/prefork.c. As this is a rather big function, and looks like it is never left while the server runs, and there are no local arrays accessed, the stack canary may be overwritten by some function called from there. But the stack canary is just checked when prefork_run exits. Kind regards, Bernhard *** stack smashing detected ***: /usr/sbin/apache2 terminated === Backtrace: = /lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7f6d8e1c11af]| 0x7f6d8e1c11af | /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f6d8e246aa7] | 0x7f6d8e246aa7 | /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x0)[0x7f6d8e246a70] | 0x7f6d8e246a70 | /usr/lib/apache2/modules/mod_mpm_prefork.so(+0x4c08)[0x7f6d8b462c08] | 0x7f6d8b462c08 | 0x7f6193a75c08: 0x7f6193a75c03 : callq 0x7f6193a73400 <__stack_chk_fail@plt> /usr/sbin/apache2(+0x29e450)[0x7f6d8f2a3450] | 0x7f6d8f2a3450 | === Memory map: 7f6d8f005000-7f6d8f09d000 r-xp fe:00 3882 /usr/sbin/apache2 7f6d8b45e000-7f6d8b465000 r-xp fe:00 127839 /usr/lib/apache2/modules/mod_mpm_prefork.so apt install dpkg-dev devscripts mc gdb binutils apache2-bin apache2-dbg # http://snapshot.debian.org/package/apache2/2.4.10-10%2Bdeb8u7/ wget http://snapshot.debian.org/archive/debian/20160916T101556Z/pool/main/a/apache2/apache2-bin_2.4.10-10%2Bdeb8u7_amd64.deb wget http://snapshot.debian.org/archive/debian/20160916T101556Z/pool/main/a/apache2/apache2-dbg_2.4.10-10%2Bdeb8u7_amd64.deb dpkg -i --force-depends apache2-bin_2.4.10-10+deb8u7_amd64.deb apache2-dbg_2.4.10-10+deb8u7_amd64.deb mkdir apache2/orig -p cdapache2/orig dget http://snapshot.debian.org/archive/debian/20160916T101556Z/pool/main/a/apache2/apache2_2.4.10-10%2Bdeb8u7.dsc dpkg-source -x apache2_2.4.10-10%2Bdeb8u7.dsc cd ../.. a2dismod mpm_event a2enmod mpm_prefork systemctl stop apache2 systemctl start apache2 root@debian:~# gdb -q --pid 16415 ... (gdb) set width 0 (gdb) set pagination off (gdb) directory /home/benutzer/apache2/orig/apache2-2.4.10/server Source directories searched: /home/benutzer/apache2/orig/apache2-2.4.10/server:$cdir:$cwd (gdb) b main Breakpoint 1 at 0x556c539ec940: file main.c, line 439. (gdb) disassemble prefork_run,prefork_run+3830 Dump of assembler code from 0x7f6193a74d60 to 0x7f6193a75c56: 0x7f6193a74d60 : push %r15 ... 0x7f6193a74d81 : mov%fs:0x28,%rax ; Value loaded into $rax 0x7f6193a74d8a : mov%rax,0xe8(%rsp) ; Value stored in canary ... 0x7f6193a75288 : mov0xe8(%rsp),%rbx ; Canary loaded into $rbx 0x7f6193a75290 : xor%fs:0x28,%rbx ; Canary compared to the original value 0x7f6193a75299 : mov%r13d,%eax 0x7f6193a7529c : jne0x7f6193a75c03 ... 0x7f6193a75c03 : callq 0x7f6193a73400 <__stack_chk_fail@plt> 0x7f6193a75c08 : callq 0x7f6193a73300 <__errno_location@plt> ... 0x7f6193a75c4b : jmpq 0x7f6193a75b9c 0x7f6193a75c50 : push %rbp End of assembler dump. set width 0 set pagination off directory /home/benutzer/apache2/orig/apache2-2.4.10/server b main run
Bug#840580: apache2-bin: crashes when issuing a restart while mod_cgid is enabled
Hi, On Wednesday, 12 October 2016 15:27:45 CET Brendon Baumgartner wrote: > We have a relatively busy webserver (about 1-2 million hits per day). > Recently we experienced some downtime and tracked it to mod_cgid. Once we > disabled this module, the crashes stopped. > > To induce the crash (doesn't always work), enable mod_cgid let the server > run for a bit. Then issue a restart. In the error log I would find the > information below. After the crash would occur, apache would no longer > restart or gracefully kill. I would have to kill -9 two remaining apache > processes. Once they were gone, I could start the process like normal. I could not reproduce this. Which command exactly did you use to restart the server? 'service apache2 restart' or 'apachectl restart'? Can you please try to get a more detailed backtrace, as described in /usr/ share/doc/apache2/README.backtrace . Also, besides from the crashing process it would be interesting to get backtraces from the two hanging processes. And the output from "ps -ef|apache2" from before the restart (to see processes' child/parent relationships). Thanks. Note that, since you seem to use mpm_prefork, you can use mod_cgi instead of mod_cgid if that works better. Cheers, Stefan > > > [Fri Oct 07 09:24:35.594582 2016] [core:error] [pid 25450] AH00546: no > record of generation 1 of exiting child 8814 [Fri Oct 07 09:24:35.594659 > 2016] [core:error] [pid 25450] AH00546: no record of generation 1 of > exiting child 8098 *** stack smashing detected ***: /usr/sbin/apache2 > terminated > === Backtrace: = > /lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7f6d8e1c11af] > /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f6d8e246aa7] > /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x0)[0x7f6d8e246a70] > /usr/lib/apache2/modules/mod_mpm_prefork.so(+0x4c08)[0x7f6d8b462c08] > /usr/sbin/apache2(+0x29e450)[0x7f6d8f2a3450] > === Memory map: > 7f6d7f673000-7f6d7f674000 ---p 00:00 0 > 7f6d7f674000-7f6d7fe74000 rw-p 00:00 0 > 7f6d867a8000-7f6d867ad000 r-xp fe:00 127386 > /usr/lib/apache2/modules/mod_status.so 7f6d867ad000-7f6d869ac000 ---p > 5000 fe:00 127386 > /usr/lib/apache2/modules/mod_status.so 7f6d869ac000-7f6d869ad000 r--p > 4000 fe:00 127386 > /usr/lib/apache2/modules/mod_status.so 7f6d869ad000-7f6d869ae000 rw-p > 5000 fe:00 127386 > /usr/lib/apache2/modules/mod_status.so 7f6d869ae000-7f6d869df000 r-xp > fe:00 127385 > /usr/lib/apache2/modules/mod_ssl.so 7f6d869df000-7f6d86bdf000 ---p 00031000 > fe:00 127385 /usr/lib/apache2/modules/mod_ssl.so > 7f6d86bdf000-7f6d86be1000 r--p 00031000 fe:00 127385 > /usr/lib/apache2/modules/mod_ssl.so 7f6d86be1000-7f6d86be2000 rw-p 00033000 > fe:00 127385 /usr/lib/apache2/modules/mod_ssl.so > 7f6d86be2000-7f6d86be4000 rw-p 00:00 0 > 7f6d86be4000-7f6d86c55000 rw-p 00:00 0 > 7f6d86c75000-7f6d86c87000 rw-s 00:04 230335829 > /dev/zero (deleted) 7f6d86c87000-7f6d86cbc000 r--s fe:03 2051 > /var/cache/nscd/services 7f6d86e6-7f6d86e65000 r-xp > fe:00 127793 > /usr/lib/apache2/modules/mod_socache_shmcb.so 7f6d86e65000-7f6d87064000 > ---p 5000 fe:00 127793 > /usr/lib/apache2/modules/mod_socache_shmcb.so 7f6d87064000-7f6d87066000 > r--p 4000 fe:00 127793 > /usr/lib/apache2/modules/mod_socache_shmcb.so 7f6d87066000-7f6d87067000 > rw-p 6000 fe:00 127793 > /usr/lib/apache2/modules/mod_socache_shmcb.so 7f6d87067000-7f6d8706a000 > r-xp fe:00 127658 > /usr/lib/apache2/modules/mod_setenvif.so 7f6d8706a000-7f6d87269000 ---p > 3000 fe:00 127658 > /usr/lib/apache2/modules/mod_setenvif.so 7f6d87269000-7f6d8726a000 r--p > 2000 fe:00 127658 > /usr/lib/apache2/modules/mod_setenvif.so 7f6d8726a000-7f6d8726b000 rw-p > 3000 fe:00 127658 > /usr/lib/apache2/modules/mod_setenvif.so 7f6d8726b000-7f6d8727b000 r-xp > fe:00 127579 > /usr/lib/apache2/modules/mod_rewrite.so 7f6d8727b000-7f6d8747a000 ---p > 0001 fe:00 127579 > /usr/lib/apache2/modules/mod_rewrite.so 7f6d8747a000-7f6d8747b000 r--p > f000 fe:00 127579 > /usr/lib/apache2/modules/mod_rewrite.so 7f6d8747b000-7f6d8747c000 rw-p > 0001 fe:00 127579 > /usr/lib/apache2/modules/mod_rewrite.so 7f6d8747c000-7f6d8747f000 r-xp > fe:00 121863 > /lib/x86_64-linux-gnu/libkeyutils.so.1.5 7f6d8747f000-7f6d8767e000 ---p > 3000 fe:00 121863 > /lib/x86_64-linux-gnu/libkeyutils.so.1.5 7f6d8767e000-7f6d8767f000 r--p > 2000 fe:00 121863
Bug#840580: apache2-bin: crashes when issuing a restart while mod_cgid is enabled
Package: apache2-bin Version: 2.4.10-10+deb8u7 Severity: important Hello. We have a relatively busy webserver (about 1-2 million hits per day). Recently we experienced some downtime and tracked it to mod_cgid. Once we disabled this module, the crashes stopped. To induce the crash (doesn't always work), enable mod_cgid let the server run for a bit. Then issue a restart. In the error log I would find the information below. After the crash would occur, apache would no longer restart or gracefully kill. I would have to kill -9 two remaining apache processes. Once they were gone, I could start the process like normal. [Fri Oct 07 09:24:35.594582 2016] [core:error] [pid 25450] AH00546: no record of generation 1 of exiting child 8814 [Fri Oct 07 09:24:35.594659 2016] [core:error] [pid 25450] AH00546: no record of generation 1 of exiting child 8098 *** stack smashing detected ***: /usr/sbin/apache2 terminated === Backtrace: = /lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7f6d8e1c11af] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f6d8e246aa7] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x0)[0x7f6d8e246a70] /usr/lib/apache2/modules/mod_mpm_prefork.so(+0x4c08)[0x7f6d8b462c08] /usr/sbin/apache2(+0x29e450)[0x7f6d8f2a3450] === Memory map: 7f6d7f673000-7f6d7f674000 ---p 00:00 0 7f6d7f674000-7f6d7fe74000 rw-p 00:00 0 7f6d867a8000-7f6d867ad000 r-xp fe:00 127386 /usr/lib/apache2/modules/mod_status.so 7f6d867ad000-7f6d869ac000 ---p 5000 fe:00 127386 /usr/lib/apache2/modules/mod_status.so 7f6d869ac000-7f6d869ad000 r--p 4000 fe:00 127386 /usr/lib/apache2/modules/mod_status.so 7f6d869ad000-7f6d869ae000 rw-p 5000 fe:00 127386 /usr/lib/apache2/modules/mod_status.so 7f6d869ae000-7f6d869df000 r-xp fe:00 127385 /usr/lib/apache2/modules/mod_ssl.so 7f6d869df000-7f6d86bdf000 ---p 00031000 fe:00 127385 /usr/lib/apache2/modules/mod_ssl.so 7f6d86bdf000-7f6d86be1000 r--p 00031000 fe:00 127385 /usr/lib/apache2/modules/mod_ssl.so 7f6d86be1000-7f6d86be2000 rw-p 00033000 fe:00 127385 /usr/lib/apache2/modules/mod_ssl.so 7f6d86be2000-7f6d86be4000 rw-p 00:00 0 7f6d86be4000-7f6d86c55000 rw-p 00:00 0 7f6d86c75000-7f6d86c87000 rw-s 00:04 230335829 /dev/zero (deleted) 7f6d86c87000-7f6d86cbc000 r--s fe:03 2051 /var/cache/nscd/services 7f6d86e6-7f6d86e65000 r-xp fe:00 127793 /usr/lib/apache2/modules/mod_socache_shmcb.so 7f6d86e65000-7f6d87064000 ---p 5000 fe:00 127793 /usr/lib/apache2/modules/mod_socache_shmcb.so 7f6d87064000-7f6d87066000 r--p 4000 fe:00 127793 /usr/lib/apache2/modules/mod_socache_shmcb.so 7f6d87066000-7f6d87067000 rw-p 6000 fe:00 127793 /usr/lib/apache2/modules/mod_socache_shmcb.so 7f6d87067000-7f6d8706a000 r-xp fe:00 127658 /usr/lib/apache2/modules/mod_setenvif.so 7f6d8706a000-7f6d87269000 ---p 3000 fe:00 127658 /usr/lib/apache2/modules/mod_setenvif.so 7f6d87269000-7f6d8726a000 r--p 2000 fe:00 127658 /usr/lib/apache2/modules/mod_setenvif.so 7f6d8726a000-7f6d8726b000 rw-p 3000 fe:00 127658 /usr/lib/apache2/modules/mod_setenvif.so 7f6d8726b000-7f6d8727b000 r-xp fe:00 127579 /usr/lib/apache2/modules/mod_rewrite.so 7f6d8727b000-7f6d8747a000 ---p 0001 fe:00 127579 /usr/lib/apache2/modules/mod_rewrite.so 7f6d8747a000-7f6d8747b000 r--p f000 fe:00 127579 /usr/lib/apache2/modules/mod_rewrite.so 7f6d8747b000-7f6d8747c000 rw-p 0001 fe:00 127579 /usr/lib/apache2/modules/mod_rewrite.so 7f6d8747c000-7f6d8747f000 r-xp fe:00 121863 /lib/x86_64-linux-gnu/libkeyutils.so.1.5 7f6d8747f000-7f6d8767e000 ---p 3000 fe:00 121863 /lib/x86_64-linux-gnu/libkeyutils.so.1.5 7f6d8767e000-7f6d8767f000 r--p 2000 fe:00 121863 /lib/x86_64-linux-gnu/libkeyutils.so.1.5 7f6d8767f000-7f6d8768 rw-p 3000 fe:00 121863 /lib/x86_64-linux-gnu/libkeyutils.so.1.5 7f6d8768-7f6d8768b000 r-xp fe:00 122014 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1 7f6d8768b000-7f6d8788a000 ---p b000 fe:00 122014 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1 7f6d8788a000-7f6d8788b000 r--p a000 fe:00 122014 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1 7f6d8788b000-7f6d8788c000 rw-p b000 fe:00 122014 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1 7f6d8788c000-7f6d878ae000 r-xp fe:00 123865
