RE: Problem with Apache22
munmap 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL munmap(0x28c3f000,0x5000) 84893 httpdRET munmap 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL munmap(0x28c38000,0x7000) 84893 httpdRET munmap 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL munmap(0x28a0a000,0x15000) 84893 httpdRET munmap 0 84893 httpdCALL munmap(0x28a1f000,0xeb000) 84893 httpdRET munmap 0 84893 httpdCALL munmap(0x28b0a000,0x7000) 84893 httpdRET munmap 0 84893 httpdCALL munmap(0x28b11000,0x2f000) 84893 httpdRET munmap 0 84893 httpdCALL munmap(0x28b4,0xf8000) 84893 httpdRET munmap 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL munmap(0x289ef000,0xf000) 84893 httpdRET munmap 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL munmap(0x289c1000,0x8000) 84893 httpdRET munmap 0 84893 httpdCALL munmap(0x289c9000,0x26000) 84893 httpdRET munmap 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL munmap(0x289b1000,0x3000) 84893 httpdRET munmap 0 84893 httpdCALL munmap(0x289b4000,0xd000) 84893 httpdRET munmap 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL sigprocmask(0x1,0x280ba5c0,0xbfbfe4b0) 84893 httpdRET sigprocmask 0 84893 httpdCALL munmap(0x289fe000,0xc000) 84893 httpdRET munmap 0 84893 httpdCALL sigprocmask(0x3,0x280ba5d0,0) 84893 httpdRET sigprocmask 0 84893 httpdCALL close(0x13) 84893 httpdRET close 0 84893 httpdCALL close(0x5) 84893 httpdRET close 0 84893 httpdCALL close(0x4) 84893 httpdRET close 0 84893 httpdCALL exit(0) -Original Message- From: Philip M. Gollucci [mailto:[EMAIL PROTECTED] Sent: Thursday, November 01, 2007 4:29 PM To: Peter Uthoff Cc: freebsd-questions@freebsd.org Subject: Re: Problem with Apache22 Peter Uthoff wrote: 10:45:11 kernel: pid 66395 (httpd), uid 80: exited on signal 4 Oct 25 Can you provide either an strace or ktrace/kdump output from one of the children. Just attached to one let it die, then send the last 500 lines or so of the output. You probably have core dumps somewhere too -- a backtrace might be helpful. Do you threading libraries match across the board for all software (aka use ldd). sysctls: kern.sugid_coredump=1 kern.corefile=core.%N.%P also, cd /var/db/pkg ; /bin/ls -1 apache* php5* mysql* mod_* I don't believe the 'busy' message is accurate. Correct. The sad part is that these all show no errors in their logs even with them turned up to debug level Thats because the error is happening before the parent hands off the request to the child. This output might be helpful to others cat /var/db/ports
Re: Problem with Apache22
Which MPM did you use, if you didn't change it the default is prefork. how about: ldd /usr/local/sbin/httpd | egrep 'libthr|libpthread|libc_r' ldd /usr/local/libexec/mysqld | egrep 'libthr|libpthread|libc_r' ldd /usr/local/bin/php| egrep 'libthr|libpthread|libc_r' ldd /usr/local/lib/libaprutil-1.so.2 | egrep 'libthr|libpthread|libc_r' ldd /usr/local/lib/libapr-1.so.2 | egrep 'libthr|libpthread|libc_r' ldd /usr/local/lib/mysql/libmysqlclient.so.15 | \ egrep 'libthr|libpthread|libc_r' WITHOUT_THREADS=true So it looks like you don't want threads. That makes things easier as its the simpler case. At any rate, you'll want the output of all the above to match. Nothing in the ktrace/kdump jumps out at me. Are you sure it crashed ? (and you were attached to the correct httpd child) httpd -X and/or httpd -DONE_PROCESS might be helpful for that. Philip M. Gollucci ([EMAIL PROTECTED]) o:703.549.2050x206 Senior System Admin - Riderway, Inc. http://riderway.com / http://ridecharge.com 1024D/EC88A0BF 0DE5 C55C 6BF3 B235 2DAB B89E 1324 9B4F EC88 A0BF Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Problem with Apache22
Yes, it's using prefork. I made no changes from the default install. The results of ldd for each of these yielded libpthread.so.2 = /lib/libpthread.so.2 in each case except for php and libmysqlclient.so.15, which both returned nothing. Running apache in debug, attached to the console resulted in output of Bus Error and nothing else. The web logs showed no errors at all. -Original Message- From: Philip M. Gollucci [mailto:[EMAIL PROTECTED] Sent: Monday, November 05, 2007 3:42 PM To: Peter Uthoff Cc: Philip M. Gollucci; freebsd-questions@freebsd.org Subject: Re: Problem with Apache22 Which MPM did you use, if you didn't change it the default is prefork. how about: ldd /usr/local/sbin/httpd | egrep 'libthr|libpthread|libc_r' ldd /usr/local/libexec/mysqld | egrep 'libthr|libpthread|libc_r' ldd /usr/local/bin/php| egrep 'libthr|libpthread|libc_r' ldd /usr/local/lib/libaprutil-1.so.2 | egrep 'libthr|libpthread|libc_r' ldd /usr/local/lib/libapr-1.so.2 | egrep 'libthr|libpthread|libc_r' ldd /usr/local/lib/mysql/libmysqlclient.so.15 | \ egrep 'libthr|libpthread|libc_r' WITHOUT_THREADS=true So it looks like you don't want threads. That makes things easier as its the simpler case. At any rate, you'll want the output of all the above to match. Nothing in the ktrace/kdump jumps out at me. Are you sure it crashed ? (and you were attached to the correct httpd child) httpd -X and/or httpd -DONE_PROCESS might be helpful for that. Philip M. Gollucci ([EMAIL PROTECTED]) o:703.549.2050x206 Senior System Admin - Riderway, Inc. http://riderway.com / http://ridecharge.com 1024D/EC88A0BF 0DE5 C55C 6BF3 B235 2DAB B89E 1324 9B4F EC88 A0BF Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. This email was sent to you by Reuters, the global news and information company. To find out more about Reuters visit www.about.reuters.com Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited. Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United Kingdom Registered No: 3296375 Registered in England and Wales ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Problem with Apache22
Peter Uthoff wrote: 10:45:11 kernel: pid 66395 (httpd), uid 80: exited on signal 4 Oct 25 Can you provide either an strace or ktrace/kdump output from one of the children. Just attached to one let it die, then send the last 500 lines or so of the output. You probably have core dumps somewhere too -- a backtrace might be helpful. Do you threading libraries match across the board for all software (aka use ldd). sysctls: kern.sugid_coredump=1 kern.corefile=core.%N.%P also, cd /var/db/pkg ; /bin/ls -1 apache* php5* mysql* mod_* I don't believe the 'busy' message is accurate. Correct. The sad part is that these all show no errors in their logs even with them turned up to debug level Thats because the error is happening before the parent hands off the request to the child. This output might be helpful to others cat /var/db/ports/apache22/options Finally, Some snippets from the httpd.conf such as loaded modules and other custom/non default things you have as well. Philip M. Gollucci ([EMAIL PROTECTED]) o:703.749.9295x206 Senior System Admin - Riderway, Inc. http://riderway.com / http://ridecharge.com 1024D/EC88A0BF 0DE5 C55C 6BF3 B235 2DAB B89E 1324 9B4F EC88 A0BF Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Problem with Apache22
On 11/1/07, Peter Uthoff [EMAIL PROTECTED] wrote: Hello, I have a problem where my Apache procs are dying almost exactly every ten minutes as you can from the messages and web logs below: Oct 25 10:34:44 kernel: pid 66337 (httpd), uid 80: exited on signal 4 Oct 25 10:35:33 kernel: pid 66357 (httpd), uid 80: exited on signal 4 Oct 25 10:45:11 kernel: pid 66395 (httpd), uid 80: exited on signal 4 Oct 25 10:55:21 kernel: pid 66340 (httpd), uid 80: exited on signal 4 Looking at the Apache logs, I find the following: [Thu Oct 25 10:25:01 2007] [notice] child pid 66379 exit signal Illegal instruction (4) [Thu Oct 25 10:34:44 2007] [notice] child pid 66337 exit signal Illegal instruction (4) [Thu Oct 25 10:35:33 2007] [notice] child pid 66357 exit signal Illegal instruction (4) [Thu Oct 25 10:45:11 2007] [notice] child pid 66395 exit signal Illegal instruction (4) I tried upping the logs all the way to debug today and it really wasn't very helpful: [Wed Oct 31 15:59:19 2007] [debug] prefork.c(991): AcceptMutex: flock (default: flock) [Wed Oct 31 16:05:39 2007] [notice] child pid 73668 exit signal Illegal instruction (4) [Wed Oct 31 16:12:01 2007] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 4 idle, and 17 total children [Wed Oct 31 16:15:04 2007] [notice] child pid 73779 exit signal Illegal instruction (4) [Wed Oct 31 16:15:28 2007] [notice] child pid 73717 exit signal Illegal instruction (4) [Wed Oct 31 16:18:37 2007] [notice] child pid 95939 exit signal Illegal instruction (4) I don't believe the 'busy' message is accurate. I think it's a result of the procs dying constantly as I just don't get that much traffic. I also have no idea what is causing the illegal instructions because none of the logs point out any specific detail to help me track it down. I'm running the following: FreeBSD 6.2-RELEASE-p8 FreeBSD 6.2-RELEASE-p8 #3: Thu Oct 25 20:04:14 CDT 2007 :/usr/obj/usr/src/sys/GENERIC i386 apache-2.2.6_2 Version 2.2 of Apache web server with prefork MPM. php5-5.2.4_1PHP Scripting Language mysql-client-5.0.45_1 Multithreaded SQL database (client) mysql-server-5.0.45_1 Multithreaded SQL database (server) p5-DBD-mysql50-4.005 MySQL 5.0 driver for the Perl5 Database Interface (DBI) php5-mysql-5.2.4_1 The mysql shared extension for php php5-mysqli-5.2.4_1 The mysqli shared extension for php I'm using vhosts to host 4 different websites with different domains. The sad part is that these all show no errors in their logs even with them turned up to debug level. I've been looking at this off and on trying to solve it as time allowed since mid-September. I'm completely stumped and would appreciate any helpful suggestions where else I can look. Thank you! This email was sent to you by Reuters, the global news and information company. To find out more about Reuters visit www.about.reuters.com Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited. Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United Kingdom Registered No: 3296375 Registered in England and Wales ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] First thing that comes to mind is to download knoppix and use either memtest or memtest86 as the cheatcode. I forget which. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Problem with Apache22
Peter Uthoff wrote: Hello, I have a problem where my Apache procs are dying almost exactly every ten minutes as you can from the messages and web logs below: Oct 25 10:34:44 kernel: pid 66337 (httpd), uid 80: exited on signal 4 Signal 4 is illegal instruction, it might be caused by: - corrupted executables - executables built for processor architectures different than the one used in the machine (e.g. built on one machine with CPU optimizations turned on, then run on another) The sad part is that these all show no errors in their logs even with them turned up to debug level. I've been looking at this off and on trying to solve it as time allowed since mid-September. I'm completely stumped and would appreciate any helpful suggestions where else I can look. Do you have any processor-specific instructions in your make.conf? How did you install apache and other applications? Have you tried recompiling apache and php without any optimizations? Are you using PHP as mod_php? If so, you might want to try to switch to fastcgi to isolate if the problem is in apache or php. signature.asc Description: OpenPGP digital signature
