DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG· RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT <http://issues.apache.org/bugzilla/show_bug.cgi?id=39744>. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND· INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=39744 Summary: SIGBUS on FreeBSD 4.x (semi-reproducable) Product: Apache httpd-2 Version: 2.2.2 Platform: Other OS/Version: FreeBSD Status: NEW Severity: normal Priority: P2 Component: mod_ssl AssignedTo: [email protected] ReportedBy: [EMAIL PROTECTED] This isn't going to be fun to explain. FreeBSD pentarou.parodius.com 4.11-STABLE FreeBSD 4.11-STABLE #0: Sun May 28 12:02:42 PDT 2006 [EMAIL PROTECTED]:/usr/o bj/usr/src/sys/PENTAROU i386 Situation: Occasionally our production Apache server dumps core on a SIGBUS: pid 31008 (httpd), uid 0: exited on signal 10 (core dumped) The time of the problem is very suspicious: a few seconds after midnight, which is when our newsyslog is scheduled to run for Apache logfiles. Note that we send Apache a signal 30, which is SIGUSR1: /var/log/httpd-access.log 640 13 * @T00 B /var/run/httpd.pid 30 /var/log/httpd-error.log 640 13 * @T00 B /var/run/httpd.pid 30 /var/log/httpd-cgiwrap.log 640 13 * @T00 Z /var/log/suphp.log 640 13 * @T00 Z Apache error_log says: [Wed Jun 07 00:00:02 2006] [notice] Graceful restart requested, doing restart Then the log rotates. New log says: [Wed Jun 07 00:00:02 2006] [notice] mod_bw : Memory Allocated 112 bytes (each conf takes 28 bytes) [Wed Jun 07 00:00:02 2006] [notice] mod_bw : Version 0.7 - Initialized [4 Confs] [Wed Jun 07 00:00:03 2006] [notice] seg fault or similar nasty error detected in the parent process Where things get crazy: the core backtrace shows a problem that seems to point to the FreeBSD crypto library. Our system libraries *are not* built with debugging symbols (but Apache is). We have one SSL virtualhost (support.parodius.com). Here's the bt full: pentarou# gdb /usr/local/sbin/httpd httpd.31008.uid_0.core GNU gdb 4.18 (FreeBSD) Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dwarf2read.c line 3049 in dwarf2_read_section Core was generated by `httpd'. Program terminated with signal 10, Bus error. Reading symbols from /usr/lib/libm.so.2...done. Reading symbols from /usr/local/lib/libaprutil-1.so.2...done. Reading symbols from /usr/local/lib/libexpat.so.6...done. Reading symbols from /usr/local/lib/libiconv.so.3...done. Reading symbols from /usr/local/lib/libapr-1.so.2...done. Reading symbols from /usr/lib/libcrypt.so.2...done. Reading symbols from /usr/lib/libc.so.4...done. Reading symbols from /usr/local/libexec/apache22/mod_authn_file.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authn_dbm.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authn_anon.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authn_default.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authz_host.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authz_groupfile.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authz_user.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authz_dbm.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authz_owner.so...done. Reading symbols from /usr/local/libexec/apache22/mod_authz_default.so...done. Reading symbols from /usr/local/libexec/apache22/mod_auth_basic.so...done. Reading symbols from /usr/local/libexec/apache22/mod_include.so...done. Reading symbols from /usr/local/libexec/apache22/mod_filter.so...done. Reading symbols from /usr/local/libexec/apache22/mod_charset_lite.so...done. Reading symbols from /usr/local/libexec/apache22/mod_deflate.so...done. Reading symbols from /usr/lib/libz.so.2...done. Reading symbols from /usr/local/libexec/apache22/mod_log_config.so...done. Reading symbols from /usr/local/libexec/apache22/mod_logio.so...done. Reading symbols from /usr/local/libexec/apache22/mod_env.so...done. Reading symbols from /usr/local/libexec/apache22/mod_mime_magic.so...done. Reading symbols from /usr/local/libexec/apache22/mod_expires.so...done. Reading symbols from /usr/local/libexec/apache22/mod_headers.so...done. Reading symbols from /usr/local/libexec/apache22/mod_usertrack.so...done. Reading symbols from /usr/local/libexec/apache22/mod_unique_id.so...done. Reading symbols from /usr/local/libexec/apache22/mod_setenvif.so...done. Reading symbols from /usr/local/libexec/apache22/mod_version.so...done. Reading symbols from /usr/local/libexec/apache22/mod_ssl.so...done. Reading symbols from /usr/lib/libssl.so.3...done. Reading symbols from /usr/lib/libcrypto.so.3...done. Reading symbols from /usr/local/libexec/apache22/mod_mime.so...done. Reading symbols from /usr/local/libexec/apache22/mod_autoindex.so...done. Reading symbols from /usr/local/libexec/apache22/mod_cgi.so...done. Reading symbols from /usr/local/libexec/apache22/mod_vhost_alias.so...done. Reading symbols from /usr/local/libexec/apache22/mod_negotiation.so...done. Reading symbols from /usr/local/libexec/apache22/mod_dir.so...done. Reading symbols from /usr/local/libexec/apache22/mod_actions.so...done. Reading symbols from /usr/local/libexec/apache22/mod_userdir.so...done. Reading symbols from /usr/local/libexec/apache22/mod_alias.so...done. Reading symbols from /usr/local/libexec/apache22/mod_rewrite.so...done. Reading symbols from /usr/local/libexec/apache22/mod_suphp.so...done. Reading symbols from /usr/local/libexec/apache22/mod_bw.so...done. Reading symbols from /usr/libexec/ld-elf.so.1...done. #0 0x2847663c in engine_table_select () from /usr/lib/libcrypto.so.3 (gdb) bt full #0 0x2847663c in engine_table_select () from /usr/lib/libcrypto.so.3 No symbol table info available. #1 0x28454df4 in ENGINE_get_default_RSA () from /usr/lib/libcrypto.so.3 No symbol table info available. #2 0x28453209 in RSA_new_method () from /usr/lib/libcrypto.so.3 No symbol table info available. #3 0x2845309d in RSA_new () from /usr/lib/libcrypto.so.3 No symbol table info available. #4 0x2846c6f7 in RSAPrivateKey_asn1_meth () from /usr/lib/libcrypto.so.3 No symbol table info available. #5 0x28478688 in ASN1_item_ex_new () from /usr/lib/libcrypto.so.3 No symbol table info available. #6 0x284784cf in ASN1_item_ex_new () from /usr/lib/libcrypto.so.3 No symbol table info available. #7 0x28474361 in ASN1_item_ex_d2i () from /usr/lib/libcrypto.so.3 No symbol table info available. #8 0x28473e12 in ASN1_item_d2i () from /usr/lib/libcrypto.so.3 No symbol table info available. #9 0x2846c7ab in d2i_RSAPublicKey () from /usr/lib/libcrypto.so.3 No symbol table info available. #10 0x28467bfe in d2i_PublicKey () from /usr/lib/libcrypto.so.3 No symbol table info available. #11 0x284666be in X509_PUBKEY_get () from /usr/lib/libcrypto.so.3 No symbol table info available. #12 0x284656e9 in X509_get_pubkey () from /usr/lib/libcrypto.so.3 No symbol table info available. #13 0x28380183 in ssl_util_algotypeof (pCert=0x830b200, pKey=0x0) at ssl_util.c:144 t = 0 pFreeKey = (struct evp_pkey_st *) 0x0 #14 0x28374e41 in ssl_pphrase_Handle (s=0x80bbeb8, p=0x8146018) at ssl_engine_pphrase.c:216 mc = (struct {...} *) 0x80b8320 sc = (struct SSLSrvConfigRec *) 0x8299378 pServ = (struct server_rec *) 0x81ada90 cpVHostID = 0x81470c0 "support.parodius.com:443" szPath = "/conf/ME/certificates/support.parodius.com.crt\000ions\n\000 \001\000\000 \211)\b\200\211)\b(ÿ¿\000\000\000\000`.)\b\000-conference/x- cooltalk\t\tice\000\000ltimedia\000\000\000ml\000\000eb\000\000ge+xml\000\000 dmg\000\000.\000\000media type\000\000\000\000\211\000\000\000L\001\000\000 \002 \000\000\024\001\000\000\203\001\000\000N\000\000\000\b\000\000\000 \000\000 \000\202\000\000\000\207\000\000\000\020\000\000\000"... pPrivateKey = (struct evp_pkey_st *) 0x65747962 asn1 = (struct {...} *) 0x20383220 ucp = (unsigned char *) 0x73656b61 <Address 0x73656b61 out of bounds> length = 1948254218 pX509Cert = (struct x509_st *) 0x830b200 bReadable = 1866670132 aPassPhrase = (struct apr_array_header_t *) 0x81470a0 nPassPhrase = 0 nPassPhraseCur = 1769236846 cpPassPhraseCur = 0x49202d20 <Address 0x49202d20 out of bounds> nPassPhraseRetry = 925773856 nPassPhraseDialog = 0 nPassPhraseDialogCur = 1919243808 bPassPhraseDialogOnce = 975206242 cpp = (char **) 0x5f646f6d i = 0 j = 0 algoCert = 0 algoKey = 0 at = 842021424 an = 0x303a3030 <Address 0x303a3030 out of bounds> cp = 0x20373020 <Address 0x20373020 out of bounds> pkey_mtime = 0 isterm = 1 rv = 0 #15 0x2836bbe1 in ssl_init_Module (p=0x80ba018, plog=0x80e8018, ptemp=0x8146018, base_server=0x80bbeb8) at ssl_engine_init.c:261 mc = (struct {...} *) 0x80b8320 sc = (struct SSLSrvConfigRec *) 0x8152f68 s = (struct server_rec *) 0x0 #16 0x807602d in ap_run_post_config (pconf=0x80ba018, plog=0x80e8018, ptemp=0x8146018, s=0x80bbeb8) at config.c:93 pHook = (struct ap_LINK_post_config_t *) 0x829a480 n = 9 rv = 0 #17 0x805f95f in main (argc=1, argv=0xbfbffc08) at main.c:706 c = 0 '\000' configtestonly = 0 confname = 0x8096599 "etc/apache22/httpd.conf" def_server_root = 0x80965b1 "/usr/local" temp_error_log = 0x0 error = 0x0 process = (struct process_rec *) 0x80b8098 server_conf = (struct server_rec *) 0x80bbeb8 pglobal = (struct apr_pool_t *) 0x80b8018 pconf = (struct apr_pool_t *) 0x80ba018 plog = (struct apr_pool_t *) 0x80e8018 ptemp = (struct apr_pool_t *) 0x8146018 pcommands = (struct apr_pool_t *) 0x80bc018 opt = (struct apr_getopt_t *) 0x80bc0b0 rv = 0 mod = (struct module_struct **) 0x80a367c optarg = 0xbfbffbac "" signal_server = (int (*)(int *, struct apr_pool_t *)) 0 There was mention of this problem (or similar) on freebsd-ports a few weeks ago, and the claim was that php5-openssl was causing the problem. This doesn't seem to be the case -- we build PHP as a CGI, and we _do not_ use php5-openssl: apache-2.2.2 Version 2.2 of Apache web server with prefork MPM. cgiwrap-3.9_3 Securely execute Web CGI scripts php5-5.1.4 PHP Scripting Language (Apache Module and CLI) php5-gd-5.1.4 The gd shared extension for php php5-iconv-5.1.4 The iconv shared extension for php php5-mbstring-5.1.4 The mbstring shared extension for php php5-mysql-5.1.4 The mysql shared extension for php php5-pcre-5.1.4 The pcre shared extension for php php5-session-5.1.4 The session shared extension for php php5-sqlite-5.1.4 The sqlite shared extension for php php5-xml-5.1.4 The xml shared extension for php php5-zlib-5.1.4 The zlib shared extension for php suphp-0.5.2_2 Securely execute PHP scripts I can provide the coredump if need be, although I'm not just going to toss it out to anyone who asks for it, since it's a setuid 0 process which sig10'd and the core contains some sensitive information. I say this is "semi"-reproducable because this is the 2nd or 3rd time I've seen it happen. I can't get it to happen every time/day. I'm left wondering if the fact that I'm sending SIGUSR1 to Apache -- twice very quickly -- due to the log rotation stuff is what's causing the problem. Let me know if you have some ideas where I should look, or additional variables I should print out via `p' that'll help in the debugging process. If you're *absolutely sure* it's a FreeBSD problem, then I'll take it up on the FreeBSD crypto/secure list. -- Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
