Hi Perl Version: Any version after 5.005 (Tested on 5.8.0) mod_perl Version: 1.27 Apache Version: 1.3.27
With the assistance of IBM we may have highlighted a possible bug within mod_perl/perl. The problem started as we could not load the IBM Websphere 5 Apache plugin if mod_perl was enabled. (On Solaris or Reh-Hat Linux) We narrowed this down to any mod_perl built against perl after 5.005. IBM support assisted us and we managed to enable a truss report that clearly shows the following. ----------------------------------------------------- Apache server calls WAS plug-in as_init() function twice. In as_init(), plug-in first checks if a WAS environment variable has been set. In order to avoid being initialized twice, when as_init() is first called, plug-in sets the environment variable, and skips the rest initialization. When as_init() is called again, WAS plug-in will be initialized. In this customer's case, the WAS environment set by WAS plug-in was wiped out by mod_perl. Therefore, when as_init() was called for the second time, plug-in initialization was skipped again. Since plug-in was not initialized correctly, pthread mutex was not created, plug-in log was not opened, and later crash happened when request was passed to plug-in. If mod_perl had not incorrectly freed the storage that WAS plug-in used for an environment variable, WAS plug-in initialization would have completed successfully and the crash on first request would not have occurred. Following are related information from truss output: -> ap_init_modules(0x7f4a0, 0x7f4c8, 0x7c64c, 0xff23c004) -> mod_app_server_http_eapi_new:as_init(0x7f4c8, 0x7f4a0, 0xfe02c9e0, 0x0) -> libc:getenv(0xfe072e6c, 0x0, 0x0, 0x0) <- libc:getenv() = 0 -> libc:putenv(0xd26a0, 0xd26a0, 0x0, 0xc6ebc) -> libc:malloc(0x94, 0xff23f0e8, 0x0, 0xc6e30) -> libthread:mutex_lock(0xff240600, 0x7a398, 0x0, 0xff23c004) <- libthread:mutex_lock() = 0 -> libc:_malloc_unlocked(0x94, 0x0, 0x0, 0xff23c004) -> libc:cleanfree(0x0, 0xff242844, 0x0, 0xff2427c0) <- libc:cleanfree() = 0 -> libc:t_delete(0xcf7c0, 0x0, 0xff2427c4, 0xff242844) -> libc:t_splay(0xcf7c0, 0x798c8, 0x0, 0x0) <- libc:t_splay() = 0 <- libc:t_delete() = 0xcf7c0 <- libc:_malloc_unlocked() = 0xcf7c8 -> libthread:mutex_unlock(0xff240600, 0xcf7c0, 0xff23c004, 0x98) <- libthread:mutex_unlock() = 0 <- libc:malloc() = 0xcf7c8 -> libc_psr:memcpy(0xcf7c8, 0xc6e30, 0x94, 0xc6e30) <- libc_psr:memcpy() = 0xcf7c8 <- libc:putenv() = 0 <- mod_app_server_http_eapi_new:as_init() = 0x7f4c8 -> libperl:perl_module_init(0x7f4c8, 0x7f4a0, 0xfe1730c0, 0x0) ... ... -> libperl:Perl_safesysfree(0xcf7c8, 0xcf7c8, 0x95600, 0x95600) <- libperl:Perl_safesysfree() = 0xcf7c8 ... ... <- libperl:perl_module_init() = 0x7f4c8 ... ... <- ap_init_modules() = 0x7f4a0 So cause of segfault on plug-in request is that mod_perl has freed storage that was allocated by WAS plug-in to act as flag for whether or not it has initialized so far. We do not know why mod_perl freed the storage. However, we do see the memory used by WAS plug-in has been used and freed by mod_perl. It is possible that there is a bug where mod_perl retained a reference to the storage after freeing it, and subsequently freed it again. ----------------------------------------------------- There is an addition part to the truss that might shed light on the problem. To spare you a 500Mb truss file, basically within the mod_perl_boot routines it assigns itself the memory block 0xcf7c8. It then uses this within other modperl routines, but the final line before that block of memory is used again by Websphere is :- ----------------------------------------------------- -> libperl:Perl_safesysfree(0xcf7c8, 0xcf7c8, 0x4840104, 0x0) -> libc:free(0xcf7c8, 0x0, 0x0, 0x0) -> libthread:mutex_lock(0xff240600, 0x79504, 0x0, 0xff23c004) <- libthread:mutex_lock() = 0 -> libc:_free_unlocked(0xcf7c8, 0x0, 0x0, 0xff23c004) <- libc:_free_unlocked() = 0xff2427c4 -> libthread:mutex_unlock(0xff240600, 0x0, 0x0, 0x0) <- libc:free() = 0 <- libperl:Perl_safesysfree() = 0xcf7c8 ----------------------------------------------------- You can see that the Perl_safesysfree reference's 0xcf7c8 with a potential "free" statement. Without knowing the full in's and out's of that routine, it could be that this statement is releasing the memory to the operating system. After that Websphere requests a free memory block and we see the problem that IBM have documented. *** mod_perl compile info Build Details o-----------o O-S: SunOS ssyd0045dsc 5.8 Generic_108528-23 sun4u sparc SUNW,Ultra-2 Compiler: gcc 3.1 Built against: apache 1.3.27 perl 5.8.0 perl Makefile.PL \ PERL_USELARGEFILES=0 \ INSTALLDIRS=perl \ PREFIX=/sbcimp/run/pd/apache_modules/1.3.27/mod_perl/1.29+perl- 5.8.0 \ USE_APXS=1 WITH_APXS=/sbcimp/run/pd/apache/1.3.27/bin/apxs \ EVERYTHING=1 *** Perl compile info Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration: Platform: osname=solaris, osvers=2.8, archname=sun4-solaris-thread-multi uname='sunos sldn0037dsc 5.8 generic_108528-12 sun4u sparc sunw, ultra-enterprise ' config_args='-Dprefix=/sbcimp/build/pd/perl/perl-5.8.0-test - Dprivlib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib - Darchlib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib - Dsitelib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib - Dsitearch=/sbcimp/build/pd/perl/perl-5.8.0-test/lib - Dlocincpth=/sbcimp/run/pd/gdbm/1.8.0/lib -Dcc=gcc -Dusethreads - Uinstallusrbinperl' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemultiplicity=define useperlio=define d_sfio=undef uselargefiles=undef usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags ='-D_REENTRANT -fno-strict-aliasing - I/sbcimp/run/pd/gdbm/1.8.0/lib', optimize='-O', cppflags='-D_REENTRANT -fno-strict-aliasing -I/sbcimp/run/pd/gdbm/1. 8.0/lib' ccversion='', gccversion='3.1', gccosandvers='solaris2.8' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=4 alignbytes=8, prototype=define Linker and Libraries: ld='gcc', ldflags ='' libpth=/usr/lib /usr/ccs/lib libs=-lsocket -lnsl -ldl -lm -lrt -lpthread -lc perllibs=-lsocket -lnsl -ldl -lm -lrt -lpthread -lc libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' ' cccdlflags='-fPIC', lddlflags='-G' Rgds Darren Heather UBS Invesment Bank Visit our website at http://www.ubs.com This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html