Hi

Perl Version: Any version after 5.005  (Tested on 5.8.0)
mod_perl Version: 1.27
Apache Version: 1.3.27

With the assistance of IBM we may have highlighted a possible bug within mod_perl/perl.

The problem started as we could not load the IBM Websphere 5 Apache plugin if mod_perl 
was enabled. (On Solaris or Reh-Hat Linux)  We narrowed this down to any mod_perl 
built against perl after 5.005.  IBM support assisted us and we managed to enable a 
truss report that clearly shows the following.

-----------------------------------------------------
Apache server calls WAS plug-in as_init() function twice.  In          
as_init(), plug-in first checks if a WAS environment variable has been 
set.  In order to avoid being initialized twice, when as_init() is     
first called, plug-in sets the environment variable, and skips the rest
initialization.  When as_init() is called again, WAS plug-in will be   
initialized.                                                           
                                                                       
In this customer's case, the WAS environment set by WAS plug-in was    
wiped out by mod_perl.  Therefore, when as_init() was called for the   
second time, plug-in initialization was skipped again.  Since plug-in  
was not initialized correctly, pthread mutex was not created, plug-in  
log was not opened, and later crash happened when request was passed to
plug-in.  If mod_perl had not incorrectly freed the storage that WAS   
plug-in used for an environment variable, WAS plug-in initialization   
would have completed successfully and the crash on first request would 
not have occurred.                                                     
                                                                       
Following are related information from truss output:                   
                                                                       
-> ap_init_modules(0x7f4a0, 0x7f4c8, 0x7c64c, 0xff23c004)              
  -> mod_app_server_http_eapi_new:as_init(0x7f4c8, 0x7f4a0, 0xfe02c9e0,
0x0)                                                                   



    -> libc:getenv(0xfe072e6c, 0x0, 0x0, 0x0)                          
    <- libc:getenv() = 0                                               
    -> libc:putenv(0xd26a0, 0xd26a0, 0x0, 0xc6ebc)
      
      -> libc:malloc(0x94, 0xff23f0e8, 0x0, 0xc6e30)
       -> libthread:mutex_lock(0xff240600, 0x7a398, 0x0, 0xff23c004)
       <- libthread:mutex_lock() = 0
       -> libc:_malloc_unlocked(0x94, 0x0, 0x0, 0xff23c004)
         -> libc:cleanfree(0x0, 0xff242844, 0x0, 0xff2427c0)
         <- libc:cleanfree() = 0
         -> libc:t_delete(0xcf7c0, 0x0, 0xff2427c4, 0xff242844)
           -> libc:t_splay(0xcf7c0, 0x798c8, 0x0, 0x0)
           <- libc:t_splay() = 0
         <- libc:t_delete() = 0xcf7c0
       <- libc:_malloc_unlocked() = 0xcf7c8
       -> libthread:mutex_unlock(0xff240600, 0xcf7c0, 0xff23c004, 0x98)
       <- libthread:mutex_unlock() = 0
     <- libc:malloc() = 0xcf7c8          
       -> libc_psr:memcpy(0xcf7c8, 0xc6e30, 0x94, 0xc6e30)             
       <- libc_psr:memcpy() = 0xcf7c8                                  
    <- libc:putenv() = 0                                               
  <- mod_app_server_http_eapi_new:as_init() = 0x7f4c8                  
  -> libperl:perl_module_init(0x7f4c8, 0x7f4a0, 0xfe1730c0, 0x0)       
  ... ...                                                              
          -> libperl:Perl_safesysfree(0xcf7c8, 0xcf7c8, 0x95600, 0x95600)              
                                                 
          <- libperl:Perl_safesysfree() = 0xcf7c8                      
  ... ...                                                              
  <- libperl:perl_module_init() = 0x7f4c8                              
  ... ...                                                              
<- ap_init_modules() = 0x7f4a0                                         
                                                                       
So cause of segfault on plug-in request is that mod_perl has freed     
storage that was allocated by WAS plug-in to act as flag for whether or
not it has initialized so far.  We do not know why mod_perl freed the  
storage.  However, we do see the memory used by WAS plug-in has been   
used and freed by mod_perl. It is possible that there is a bug where   
mod_perl retained a reference to the storage after freeing it, and     
subsequently freed it again.
-----------------------------------------------------

There is an addition part to the truss that might shed light on the problem.  To spare 
you a 500Mb truss file, basically within the mod_perl_boot routines it assigns itself 
the memory block 0xcf7c8.  It then uses this within other modperl routines, but the 
final line before that block of memory is used again by Websphere is :-

-----------------------------------------------------
                                            -> libperl:Perl_safesysfree(0xcf7c8, 
0xcf7c8, 0x4840104, 0x0)
                                              -> libc:free(0xcf7c8, 0x0, 0x0, 0x0)
                                                -> libthread:mutex_lock(0xff240600, 
0x79504, 0x0, 0xff23c004)
                                                <- libthread:mutex_lock() = 0
                                                -> libc:_free_unlocked(0xcf7c8, 0x0, 
0x0, 0xff23c004)
                                                <- libc:_free_unlocked() = 0xff2427c4
                                              -> libthread:mutex_unlock(0xff240600, 
0x0, 0x0, 0x0)
                                              <- libc:free() = 0
                                            <- libperl:Perl_safesysfree() = 0xcf7c8
-----------------------------------------------------

You can see that the Perl_safesysfree reference's 0xcf7c8 with a potential "free" 
statement.  Without knowing the full in's and out's of that routine, it could be that 
this statement is releasing the memory to the operating system.  After that Websphere 
requests a free memory block and we see the problem that IBM have documented.




*** mod_perl compile info                                               
Build Details                                                           
o-----------o                                                           
O-S:                                                                    
  SunOS ssyd0045dsc 5.8 Generic_108528-23 sun4u sparc SUNW,Ultra-2      
Compiler:                                                               
  gcc 3.1                                                               
Built against:                                                          
  apache   1.3.27                                                       
  perl     5.8.0                                                        
  perl Makefile.PL \                                                    
    PERL_USELARGEFILES=0 \                                              
    INSTALLDIRS=perl \                                                  
    PREFIX=/sbcimp/run/pd/apache_modules/1.3.27/mod_perl/1.29+perl-     
5.8.0 \                                                                 
    USE_APXS=1 WITH_APXS=/sbcimp/run/pd/apache/1.3.27/bin/apxs \        
    EVERYTHING=1 

                                                       
*** Perl compile info                                                   
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
  Platform:                                                             
    osname=solaris, osvers=2.8, archname=sun4-solaris-thread-multi      
    uname='sunos sldn0037dsc 5.8 generic_108528-12 sun4u sparc sunw,    
ultra-enterprise '                                                      
    config_args='-Dprefix=/sbcimp/build/pd/perl/perl-5.8.0-test -       
Dprivlib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib -                    
Darchlib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib -                    
Dsitelib=/sbcimp/build/pd/perl/perl-5.8.0-test/lib -                    
Dsitearch=/sbcimp/build/pd/perl/perl-5.8.0-test/lib -                   
Dlocincpth=/sbcimp/run/pd/gdbm/1.8.0/lib -Dcc=gcc -Dusethreads -        
Uinstallusrbinperl'                                                     
    hint=recommended, useposix=true, d_sigaction=define                 
    usethreads=define use5005threads=undef useithreads=define           
usemultiplicity=define                                                  
    useperlio=define d_sfio=undef uselargefiles=undef usesocks=undef    
    use64bitint=undef use64bitall=undef uselongdouble=undef             
    usemymalloc=n, bincompat5005=undef                                  
  Compiler:                                                             
    cc='gcc', ccflags ='-D_REENTRANT -fno-strict-aliasing -             
I/sbcimp/run/pd/gdbm/1.8.0/lib',                                        
    optimize='-O',                                                      
    cppflags='-D_REENTRANT -fno-strict-aliasing -I/sbcimp/run/pd/gdbm/1.
8.0/lib'                                                                
    ccversion='', gccversion='3.1', gccosandvers='solaris2.8'           
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321      
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',  
lseeksize=4                                                             
    alignbytes=8, prototype=define                                      
  Linker and Libraries:                                                 
    ld='gcc', ldflags =''                                               
    libpth=/usr/lib /usr/ccs/lib                                        
    libs=-lsocket -lnsl -ldl -lm -lrt -lpthread -lc                     
    perllibs=-lsocket -lnsl -ldl -lm -lrt -lpthread -lc                 
    libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a       
    gnulibc_version=''                                                  
  Dynamic Linking:                                                      
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '        
    cccdlflags='-fPIC', lddlflags='-G'

Rgds
Darren Heather
UBS Invesment Bank

Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named.  If you are not the named addressee you
should not disseminate, distribute or copy this e-mail.  Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses.  The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission.  If
verification is required please request a hard-copy version.  This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.


--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Reply via email to