Dear mod_perl experts: Collectively, we've been at this for more than two weeks and have searched various mod_perl archives, all to no avail.
Symptom: ======= SIGSEGV after fork(). Very reproducible. Memory corruption gets moved around if the codebase changes. Tried: ===== * disable rule: EXPAT * gdb * strace -f * MOD_PERL_TRACE=all ***NOTE*** Read no further unless you're interested in helping, since the rest is debug/more info. Thanks! :) Software: ======== Linux: ----- 2.4.13-ac5-20011101-1 #1 SMP Thu Nov 1 16:07:19 CST 2001 i686 unknown Apache: ------ Server version: Apache/1.3.23 (Unix) Server built: Feb 5 2002 22:16:11 Server's Module Magic Number: 19990320:11 Server compiled with.... -D HAVE_MMAP -D HAVE_SHMGET -D USE_SHMGET_SCOREBOARD -D USE_MMAP_FILES -D HAVE_FCNTL_SERIALIZED_ACCEPT -D HAVE_SYSVSEM_SERIALIZED_ACCEPT -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D HTTPD_ROOT="/home/mfister/vendor/apache-1.3.23" -D SUEXEC_BIN="/home/mfister/vendor/apache-1.3.23/sbin/suexec" -D DEFAULT_PIDLOG="www/logs/httpd.pid" -D DEFAULT_SCOREBOARD="www/logs/httpd.scoreboard" -D DEFAULT_LOCKFILE="www/logs/httpd.lock" -D DEFAULT_XFERLOG="www/logs/access_log" -D DEFAULT_ERRORLOG="www/logs/error_log" -D TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" -D ACCESS_CONFIG_FILE="conf/access.conf" -D RESOURCE_CONFIG_FILE="conf/srm.conf" Perl: ---- Summary of my perl5 (revision 5.0 version 6 subversion 1) configuration: Platform: osname=linux, osvers=2.4.13-ac5-20011101-1, archname=i686-linux uname='linux <server_name_deleted> 2.4.13-ac5-20011101-1 #1 smp thu nov 1 16:07:19 cst 2001 i686 unknown ' config_args='-des -Doptimize=-g -Dusedevel -Uinstallusrbinperl -Ubincompat5005 -Uusemymalloc -Dcc=gcc -pipe -g -I/usr/include/db3 -Dcccdlflags=-fPIC -Dinstallprefix=/home/mfister/vendor/perl-5.6.1 -Dprefix=/home/mfister/vendor/perl-5.6.1 [EMAIL PROTECTED] -Dinstallman1dir=/home/mfister/vendor/perl-5.6.1/man/man1 -Dinstallman3dir=/home/mfister/vendor/perl-5.6.1/man/man3 -Dman1dir=/home/mfister/vendor/perl-5.6.1/man/man1 -Dman3dir=/home/mfister/vendor/perl-5.6.1/man/man3 -Dd_dosuid=n -Dd_semctl_semun -Di_db -Di_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Uuselargefiles' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef useperlio=undef d_sfio=undef uselargefiles=undef usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef Compiler: cc='gcc -pipe -g -I/usr/include/db3', ccflags ='-DDEBUGGING -fno-strict-aliasing -I/usr/local/include', optimize='-g', cppflags='-DDEBUGGING -fno-strict-aliasing -I/usr/local/include' ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.1 2.96-85)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=4 alignbytes=4, usemymalloc=n, prototype=define Linker and Libraries: ld='gcc -pipe -g -I/usr/include/db3', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -ldl -lm -lc -lcrypt -lutil perllibs=-lnsl -ldl -lm -lc -lcrypt -lutil libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl.a Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic' cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: DEBUGGING Built under linux Compiled at Feb 4 2002 13:40:29 @INC: /home/mfister/vendor/perl-5.6.1/lib/5.6.1/i686-linux /home/mfister/vendor/perl-5.6.1/lib/5.6.1 /home/mfister/vendor/perl-5.6.1/lib/site_perl/5.6.1/i686-linux /home/mfister/vendor/perl-5.6.1/lib/site_perl/5.6.1 /home/mfister/vendor/perl-5.6.1/lib/site_perl mod_perl: -------- version: 1.26 built with: perl Makefile.PL USE_APXS=1 \ WITH_APXS=/home/mfister/vendor/apache-1.3.23/sbin/apxs \ EVERYTHING=1 CCFLAGS='-pipe -fPIC' Modules: ------- *** fairly long list *** Will send if necessary *** Debug Info: ========== Stack Trace: =========== #0 __pthread_mutex_lock (mutex=0x8bf04999) at mutex.c:99 #1 0x401b9cc8 in __libc_free (mem=0x4046cc18) at malloc.c:3152 #2 0x403ce028 in Perl_safesysfree (where=0x4046cc18) at util.c:158 #3 0x403f20d8 in Perl_sv_clear (sv=0x8198f60) at sv.c:3827 #4 0x403f2473 in Perl_sv_free (sv=0x8198f60) at sv.c:3950 #5 0x403f80e1 in do_clean_all (sv=0x8198f60) at sv.c:8411 #6 0x403e9c5e in S_visit (f=0x403f8094 <do_clean_all>) at sv.c:162 #7 0x403e9ce2 in Perl_sv_clean_all () at sv.c:193 #8 0x4038594a in perl_destruct (my_perl=0x809a9a8) at perl.c:665 #9 0x4035629c in perl_shutdown (s=0x0, p=0x0) at mod_perl.c:294 #10 0x40356be6 in mp_dso_unload (data=0x808e714) at mod_perl.c:489 #11 0x08050f34 in run_cleanups (c=0x809c8ac) at alloc.c:1713 #12 0x0804f5fa in ap_clear_pool (a=0x808e714) at alloc.c:538 #13 0x08062128 in standalone_main (argc=7, argv=0xbffff294) at http_main.c:5014 #14 0x08062cb2 in main (argc=7, argv=0xbffff294) at http_main.c:5401 #15 0x40155627 in __libc_start_main (main=0x80627d4 <main>, argc=7, ubp_av=0xbffff294, init=0x804e3e4 <_init>, fini=0x807aa40 <_fini>, rtld_fini=0x4000dcd4 <_dl_fini>, stack_end=0xbffff28c) at ../sysdeps/generic/libc-start.c:129 ***NOTE*** the following gdb session was gleaned from sv.c and refers to the freed memory location (0x4046cc18) above: (gdb) p *((XPV*)(sv)->sv_any) $13 = {xpv_pv = 0x4046cc18 "", xpv_cur = 135562488, xpv_len = 135617180} strace -f: --------- open("/v/01/home/mfister/ulist/run/ulist/logs/access_log", O_WRONLY|O_APPEND|O_C REAT, 0644) = 3 fcntl64(3, F_DUPFD, 15) = 17 close(3) = 0 chdir("/") = 0 fork() = 32715 [pid 32715] getpid() = 32715 [pid 32715] getrlimit(0x3, 0xbffff154) = 0 [pid 32715] setrlimit(RLIMIT_STACK, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFIN ITY}) = 0 [pid 32715] setsid() = 32715 [pid 32715] close(0) = 0 [pid 32715] open("/dev/null", O_RDONLY) = 0 [pid 32715] close(1) = 0 [pid 32715] open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1 [pid 32715] getpid() = 32715 [pid 32715] time(NULL) = 1012968984 [pid 32715] close(17) = 0 [pid 32715] close(16) = 0 [pid 32714] _exit(0) = ? rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No su ch file or directory) open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [RT_0], 8) = 0 --- SIGSEGV (Segmentation fault) --- Apache's config.status: ---------------------- CC="gcc -pipe -g -I/usr/include/db1" \ CFLAGS="-fPIC" \ LIBS="-lpthread -ldb" \ ./configure \ "--with-layout=Apache" \ "--prefix=/home/mfister/vendor/apache-1.3.23" \ "--sbindir=/home/mfister/vendor/apache-1.3.23/sbin" \ "--mandir=/home/mfister/vendor/apache-1.3.23/man" \ "--datadir=/home/mfister/vendor/apache-1.3.23/www" \ "--iconsdir=/home/mfister/vendor/apache-1.3.23/www/icons" \ "--htdocsdir=/home/mfister/vendor/apache-1.3.23/www/html" \ "--cgidir=/home/mfister/vendor/apache-1.3.23/www/cgi-bin" \ "--localstatedir=/home/mfister/vendor/apache-1.3.23/www/logs" \ "--runtimedir=/home/mfister/vendor/apache-1.3.23/www/logs" \ "--logfiledir=/home/mfister/vendor/apache-1.3.23/www/logs" \ "--proxycachedir=/home/mfister/vendor/apache-1.3.23/www/cache" \ "--enable-module=all" \ "--enable-shared=max" \ "--disable-rule=WANTHSREGEX" \ "--disable-rule=EXPAT" \ "--enable-module=auth_dbm" \ "--with-perl=/home/mfister/vendor/perl-5.6.1/bin/perl" \ MOD_PERL_TRACE=all ------------------ *** stderr of parent Apache process: perl_parse args: '/dev/null' ...allocating perl interpreter...ok constructing perl interpreter...ok ok running perl interpreter...ok mod_perl: 0 END blocks encountered during server startup loading perl module 'Apache'...loading perl module 'Apache::Constants::Exports'...ok ok mod_perl: calling perl_startup() init `PerlHandler' stack perl_cmd_push_handlers: @PerlHandler, 'AMPRPC::Server' pushing `AMPRPC::Server' into `PerlHandler' handlers init `PerlChildInitHandler' stack perl_cmd_push_handlers: @PerlChildInitHandler, 'AMPRPC::Server::handle_init' pushing `AMPRPC::Server::handle_init' into `PerlChildInitHandler' handlers init `PerlChildExitHandler' stack perl_cmd_push_handlers: @PerlChildExitHandler, 'AMPRPC::Server::handle_exit' pushing `AMPRPC::Server::handle_exit' into `PerlChildExitHandler' handlers init `PerlTransHandler' stack perl_cmd_push_handlers: @PerlTransHandler, 'ApacheRequestWatcher::start' pushing `ApacheRequestWatcher::start' into `PerlTransHandler' handlers init `PerlLogHandler' stack perl_cmd_push_handlers: @PerlLogHandler, 'ApacheRequestWatcher::stop' pushing `ApacheRequestWatcher::stop' into `PerlLogHandler' handlers perl_cmd_pass_env: arg=`LC_ALL' perl_cmd_pass_env: arg=`NLS_LANG' perl_cmd_pass_env: arg=`DN_RELBASE' perl_cmd_pass_env: arg=`PERL5LIB' perl_cmd_pass_env: arg=`SQUID_PORT' perl_cmd_pass_env: arg=`DBG' perl_cmd_pass_env: arg=`UID' perl_cmd_pass_env: arg=`USER' perl_cmd_pass_env: arg=`CONFIG_FILE' perl_cmd_pass_env: arg=`ORACLE_HOME' perl_cmd_pass_env: arg=`LD_LIBRARY_PATH' perl_cmd_pass_env: arg=`PATH' PerlRequire: arg=`AMPRPC/AMPRPCStartup.pl' attempting to require `AMPRPC/AMPRPCStartup.pl' I will use localhost:11222 to reach CCS server I will use localhost:10222 to reach IMGSUBMIT server I will use product_au:ulist/tsilu (hqtest0.deja.com/1521:UTF8) to reach ORACLE server init `PerlHandler' stack perl_cmd_push_handlers: @PerlHandler, 'Apache::Status' pushing `Apache::Status' into `PerlHandler' handlers *** stderr once error_log is stderr: loading perl module 'Apache'...ok perl_startup: perl aleady running...ok cleanup_sv: SvREFCNT(0x8128804)==1 cleanup_sv: SvREFCNT(0x80aa590)==1 cleanup_sv: SvREFCNT(0x80aa4e8)==1 cleanup_sv: SvREFCNT(0x80aa758)==1 cleanup_sv: SvREFCNT(0x80aa770)==1 cleanup_sv: SvREFCNT(0x811b328)==1 Time::HiRes dl handle == 0x812d118 POSIX dl handle == 0x813e3d8 Fcntl dl handle == 0x8231f08 Storable dl handle == 0x8221c18 HTMunger dl handle == 0x8235ed8 Data::Dumper dl handle == 0x825dbd0 Sys::Hostname dl handle == 0x8273a28 IO dl handle == 0x82909c0 DBI dl handle == 0x836e4f0 Image::Magick dl handle == 0x87c4ca8 MD5 dl handle == 0x87f3008 DBD::Oracle dl handle == 0x8cab508 running 3 END blocks for perl_shutdown destructing and freeing Perl interpreter (level=2)... -- \_/} Mark P. Fister Java, Java, everywhere, and all \_/} \_/} eBay, Inc. the cups did shrink; Java, Java \_/} \_/} Austin, TX everywhere, nor any drop to drink! \_/}