I finally found some time to get a debugger on the apache processes, and do
some load testing until one of the children sigbussed. It seems that
leaving the debugger on caused most of them to sigbus, and eventually some
children started showing sigsegv in another section of code.
I haven't had time to dig into the source yet to see where the problem lies,
but if someone wants to take a look, be my guest. I've attached 5 sigbus
traces and 1 sigsegv trace below.
It would be interesting to see if the people seeing random children die
occasionally on solaris show the same stack trace or not.
These were obtained on Apache/1.3.12 (Unix) mod_ssl/2.6.6 OpenSSL/0.9.6
running IRIX 6.5.8 with dbx attached to 25 children while under heavy (2
CPUs pegged) SSL load (using WebBench).
-Dave
(dbx) where
> 0 memcpy(0x10106fd8, 0x43cc000, 0x36, 0x0, 0x1, 0x36, 0x0, 0x36)
["/xlv47/6.5.8m/work/irix/lib/libc/libc_n32_M4/strings/bcopy.s":496,
0xfadeaac]
1 ap_bwrite(0x100e41b0, 0x43cc000, 0x36, 0x0, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/buff.c":1329,
0x10019018]
2 ap_send_mmap(0x43cc000, 0x1014b7c8, 0x0, 0x36, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_protocol.c":2332
, 0x10039614]
3 default_handler(0x1014b7c8, 0x7ffe8bd1, 0x7ffe8bd1, 0x0, 0x1, 0x36,
0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_core.c":3224,
0x10032d14]
4 ap_invoke_handler(0x1014b7c8, 0x100e3260, 0x0, 0x0, 0x1014c350, 0x0,
0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_config.c":520,
0x10025020]
5 ap_run_sub_req(0x1014b7c8, 0x43cc000, 0x36, 0x0, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_request.c":929,
0x1001be74]
6 <Unknown>() [< unknown >, 0x5fee3c48]
Process 112264 (httpd) stopped on signal SIGBUS: Bus error (handler
sig_coredump) at [memcpy:496 ,0xfadeaac]
Source (of
/xlv47/6.5.8m/work/irix/lib/libc/libc_n32_M4/strings/bcopy.s) not available
for Process 112264
(dbx) where
> 0 memcpy(0x10106fdc, 0x43cc000, 0x35, 0x8, 0x1, 0x35, 0x0, 0x35)
["/xlv47/6.5.8m/work/irix/lib/libc/libc_n32_M4/strings/bcopy.s":531,
0xfadeb30]
1 ap_bwrite(0x100e41b0, 0x43cc000, 0x35, 0x8, 0x1, 0x35, 0x0, 0x35)
["/local/share/downloads/Apache/apache_1.3.12/src/main/buff.c":1329,
0x10019018]
2 ap_send_mmap(0x43cc000, 0x1014b7c8, 0x0, 0x35, 0x1, 0x35, 0x0, 0x35)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_protocol.c":2332
, 0x10039614]
3 default_handler(0x1014b7c8, 0x7ffe8bd1, 0x7ffe8bd1, 0x8, 0x1, 0x35,
0x0, 0x35)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_core.c":3224,
0x10032d14]
4 ap_invoke_handler(0x1014b7c8, 0x100e3260, 0x0, 0x0, 0x1014c350, 0x0,
0x0, 0x35)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_config.c":520,
0x10025020]
5 ap_run_sub_req(0x1014b7c8, 0x43cc000, 0x35, 0x8, 0x1, 0x35, 0x0, 0x35)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_request.c":929,
0x1001be74]
6 <Unknown>() [< unknown >, 0x5fee3c48]
Process 444361 (httpd) stopped on signal SIGBUS: Bus error (handler
sig_coredump) at [memcpy:531 ,0xfadeb30]
Source (of
/xlv47/6.5.8m/work/irix/lib/libc/libc_n32_M4/strings/bcopy.s) not available
for Process 444361
> 0 memcpy(0x10106fd8, 0x43cc000, 0x36, 0x0, 0x1, 0x36, 0x0, 0x36)
["/xlv47/6.5.8m/work/irix/lib/libc/libc_n32_M4/strings/bcopy.s":496,
0xfadeaac]
1 ap_bwrite(0x100e41b0, 0x43cc000, 0x36, 0x0, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/buff.c":1329,
0x10019018]
2 ap_send_mmap(0x43cc000, 0x1014b7c8, 0x0, 0x36, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_protocol.c":2332
, 0x10039614]
3 default_handler(0x1014b7c8, 0x7ffe8bd1, 0x7ffe8bd1, 0x0, 0x1, 0x36,
0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_core.c":3224,
0x10032d14]
4 ap_invoke_handler(0x1014b7c8, 0x100e3260, 0x0, 0x0, 0x1014c350, 0x0,
0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_config.c":520,
0x10025020]
5 ap_run_sub_req(0x1014b7c8, 0x43cc000, 0x36, 0x0, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_request.c":929,
0x1001be74]
6 <Unknown>() [< unknown >, 0x5fee3c48]
Process 444510 (httpd) stopped on signal SIGBUS: Bus error (handler
sig_coredump) at [memcpy:496 ,0xfadeaac]
Source (of
/xlv47/6.5.8m/work/irix/lib/libc/libc_n32_M4/strings/bcopy.s) not available
for Process 444510
(dbx) where
> 0 memcpy(0x10106fd9, 0x43cc000, 0x36, 0x8, 0x1, 0x36, 0x0, 0x36)
["/xlv47/6.5.8m/work/irix/lib/libc/libc_n32_M4/strings/bcopy.s":531,
0xfadeb30]
1 ap_bwrite(0x100e41b0, 0x43cc000, 0x36, 0x8, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/buff.c":1329,
0x10019018]
2 ap_send_mmap(0x43cc000, 0x1014b7c8, 0x0, 0x36, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_protocol.c":2332
, 0x10039614]
3 default_handler(0x1014b7c8, 0x7ffe8bd1, 0x7ffe8bd1, 0x8, 0x1, 0x36,
0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_core.c":3224,
0x10032d14]
4 ap_invoke_handler(0x1014b7c8, 0x100e3260, 0x0, 0x0, 0x1014c350, 0x0,
0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_config.c":520,
0x10025020]
5 ap_run_sub_req(0x1014b7c8, 0x43cc000, 0x36, 0x8, 0x1, 0x36, 0x0, 0x36)
["/local/share/downloads/Apache/apache_1.3.12/src/main/http_request.c":929,
0x1001be74]
6 <Unknown>() [< unknown >, 0x5fee3c48]
Process 447919 (httpd) stopped on signal SIGBUS: Bus error (handler
sig_coredump) at [memcpy:531 ,0xfadeb30]
Source (of
/xlv47/6.5.8m/work/irix/lib/libc/libc_n32_M4/strings/bcopy.s) not available
for Process 447919
(dbx) where
> 0 table_adjust(0x42a4438, 0x6fe, 0x6fe, 0xd1aae1fd, 0x6fe, 0x8c,
0x42cf6e0, 0x23)
["/local/share/downloads/Apache/apache_1.3.12/src/modules/ssl/ssl_util_table
.c":1773, 0x417f9b0]
1 table_delete(0x42a4438, 0x435e374, 0x20, 0x0, 0x0, 0x8c, 0x0, 0x23)
["/local/share/downloads/Apache/apache_1.3.12/src/modules/ssl/ssl_util_table
.c":1542, 0x417f3a4]
2 ssl_scache_shm_expire(0x100ffca0, 0x39db9df7, 0x435e374, 0xc96,
0xb0e71893, 0x10102e78, 0x0, 0x23)
["/local/share/downloads/Apache/apache_1.3.12/src/modules/ssl/ssl_engine_sca
che.c":869, 0x41733e8]
3 ssl_scache_expire(0x100ffca0, 0x39db9df7, 0x4dc8adfd, 0xd1aae1fd,
0xb0e71893, 0x8c, 0x0, 0x23)
["/local/share/downloads/Apache/apache_1.3.12/src/modules/ssl/ssl_engine_sca
che.c":238, 0x4171530]
4 ssl_scache_store(0x100ffca0, 0x1035b5b0, 0x39dba04f, 0xd1aae1fd, 0x0,
0x10, 0x1035b5f8, 0x23)
["/local/share/downloads/Apache/apache_1.3.12/src/modules/ssl/ssl_engine_sca
che.c":144, 0x41711dc]
5 ssl_callback_NewSessionCacheEntry(0x100fa250, 0x1035b5b0, 0x4dc8adfd,
0xd1aae1fd, 0xb0e71893, 0x8c, 0x0, 0x23)
["/local/share/downloads/Apache/apache_1.3.12/src/modules/ssl/ssl_engine_ker
nel.c":1711, 0x416c4dc]
6 <Unknown>() [< unknown >, 0x41855fc]
Process 448119 (httpd) stopped on signal SIGSEGV: Segmentation violation
(handler sig_coredump) at [table_adjust:1773 +0x10,0x417f9b0]
Source (of
/local/share/downloads/Apache/apache_1.3.12/src/modules/ssl/ssl_util_table.c
) not available for Process 448119
______________________________________________________________________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
