hi all,
here's a problem it seems a lot of people are seeing when running a CGI
script
many people, including myself have had an ongoing problem with the
seemingly "unfindable" shared object library libclntsh.so.x.x (mine is
libclntsh.so.10.1) needed for connecting to a remote oracle database.
after installing the the sqlplus client (instantclient_10_2) on my linux
machine (and testing it on the command line, successfully connecting to
our oracle db), and then installing the perl module(s) needed to make
connections in perl scripts and cgis (DBD-Oracle-1.19), i find, like so
many others i've seen complain about this, that my cgi script runs fine
from the command line, but fails when it is run as a CGI via my
machine's apache webserver.
despite all the confident sermons about unset/mis-set environment
variables under apache (LD_LIBRARY_PATH, ORACLE_HOME, etc, etc) being
the problem (as one would first suspect, granted), changing these
settings does not fix this problem at all. i have played for hours with
these settings in BEGIN statements in the cgi, in the
/etc/sysconfig/httpd file, running httpd directly from the command line
with these env vars set in the current shell, and they DO show up when i
print out the @ENV array in the cgi, but to no avail in solving or
tracking down the problem. finally, i found one source which had more
ideas about how to analyze this problem:
http://www.mail-archive.com/dbi-users@perl.org/msg28109.html
by Philip Garrett. setting the env var
LD_DEBUG=libs
and then running httpd from the command line with the -X option allowed
me to see all the dynamic library searching and loading that was going
on behind the scenes. after adding the necessary "PassEnv LD_DEBUG"
directive to the apache config file, i can now see all the dynamic lib
loading, including that done for the CGI script via the web server.
what i am finding is that the library IS actually being found, but is
just being ignored as if it were not there.
i started by making sure libclntsh.so.10.1 could be found in the
/etc/ld.so.cache by adding the line
/usr/local/instantclient_10_2 to /etc/ld.so.conf and running ldconfig.
sure enough, it is now listed in the cache:
$ ldconfig -p | grep libclntsh
libclntsh.so.10.1 (libc6) =>
/usr/local/instantclient_10_2/libclntsh.so.10.1
libclntsh.so (libc6) => /usr/local/instantclient_10_2/libclntsh.so
(/usr/local/libclntsh.so is a link to /usr/local/libclntsh.so.10.1 in
case this matters)
now, i run my script:
#!/usr/bin/perl -w
use Carp;
use diagnostics;
use DBI;
#BEGIN {
#$ENV{'LD_LIBRARY_PATH'} = "/usr/local/instantclient_10_2";
#$ENV{'ORACLE_HOME'} = "/usr/local/instantclient_10_2";
#}
print "Content-type: text/html\n\n";
# print environment
foreach my $key (keys(%ENV)) {
if ($key =~ /^(LD|ORACLE)/) {
print "$key = $ENV{$key}<BR>\n";
}
}
print "<BR><BR>\n";
my $ans = DBI->connect("DBI:Oracle:$ENV{'ORACLE_SID'}",
'scott','tiger',
{ RaiseError => 1,
PrintError => 0,
AutoCommit => 0, } ) || print "connection
failed\n" ;
print "ans = $ans\n\n";
$ans->disconnect;
from the command line (i unset LD_LIBRARY_PATH and ORACLE_HOME to show
they are not needed or used in any way here.)
% unsetenv LD_DEBUG
% unsetenv LD_LIBRARY_PATH
% unsetenv ORACLE_HOME
% perl test2.pl
Content-type: text/html
ORACLE_SID = epgy<BR>
<BR><BR>
ans = DBI::db=HASH(0x867b0d0)
the last line shows that the connect to the oracle database worked
fine. ($ans is the handle to the oracle db.) running again, tracing the
lib loading:
% setenv LD_DEBUG libs
% perl test2.pl
14505: find library=libperl.so [0]; searching (and lots more
lines until...)
.
.
.
14505:
14505: initialize program: perl
14505:
14505:
14505: transferring control: perl
14505:
14505:
14505: calling init:
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/DBI/DBI.so
14505:
Content-type: text/html
LD_DEBUG = libs<BR>
ORACLE_SID = epgy<BR>
<BR><BR>
14505: find library=libclntsh.so.10.1 [0]; searching
14505: search
path=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE (RPATH from file
perl)
14505: trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libclntsh.so.10.1
14505: search cache=/etc/ld.so.cache
14505: trying
file=/usr/local/instantclient_10_2/libclntsh.so.10.1 (((NOTE: Found
it!)))
14505:
14505: find library=libnnz10.so [0]; searching
.
.
.
14505: calling init:
/usr/local/instantclient_10_2/libclntsh.so.10.1
14505:
14505:
14505: calling init:
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/DBD/Oracle/Oracle.so
.
.
.
ans = DBI::db=HASH(0x8b482dc)
.
.
.
again, we see that the connect has worked, after the libclntsh.so.10.1
was found and init'd successfully earlier in the process.
now i will run this script as a CGI via the web server and look in the
web server's error log to watch the lib loading debug messages. first
as root, i start the webserver in debug mode:
$ /sbin/service httpd stop
Stopping httpd: [FAILED]
$ setenv LD_DEBUG libs
$ unsetenv LD_LIBRARY_PATH
$ unsetenv ORACLE_HOME
$ /usr/sbin/httpd -X > & capture_file
here are some bits from the capture file:
[Thu Aug 30 14:20:43 2007] [notice] core dump file size limit raised to
4294967295 bytes
[Thu Aug 30 14:20:44 2007] [notice] suEXEC mechanism enabled (wrapper:
/usr/sbin/suexec)
14589:
14589: calling fini: /etc/httpd/modules/mod_ssl.so [0]
14589:
.
.
.
14589: find library=libperl.so [0]; searching
14589: search
path=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE (RPATH
from file /etc/httpd/modules/mod_perl.so)
14589: trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so
14589:
14589: find library=libutil.so.1 [0]; searching
14589: search
path=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE (RPATH
from file /etc/httpd/modules/mod_perl.so)
14589: trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libutil.so.1
14589: search cache=/etc/ld.so.cache
14589: trying file=/lib/libutil.so.1
.
.
.
14589: calling init:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so
14589:
14589:
14589: calling init: /etc/httpd/modules/mod_perl.so
.
.
.
14589: calling init: /usr/lib/php4/mysql.so
14589:
[Thu Aug 30 14:20:45 2007] [notice] mod_python: Creating 4 session
mutexes based on 150 max processes and 0 max threads.
14589: /lib/libnss_files.so.2: error: symbol lookup error:
undefined symbol: _nss_files_initgroups_dyn (fatal)
that's it at the startup of httpd (i'm not sure what that last line is
about, but i don't think it is involved in the issue here. anyone?) no
sign of loading libclntsh.so.10.1 yet, but the CGI script which needs it
has not yet been called...
(btw, the capture_file won't show the trace when i hit the cgi. but the
trace appears in the httpd error_log.)
here we go, when i load the CGI in my browser, here's the lib load trace
(this is a mess to look at, but see the lines in red for the most (i
think) important information...)
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\tfind
library=libperl.so [0]; searching
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
search path=
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/tls/i686/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
:/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/tls/i686
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
:/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/tls/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
:/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/tls
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
:/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/i686/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
:/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/i686
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
:/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
:/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] \t\t(RPATH from
file /usr/bin/perl)
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/tls/i686/sse2/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/tls/i686/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/tls/sse2/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/tls/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/i686/sse2/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/i686/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/sse2/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
.
.
.
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
14642:\tcalling init:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
14642:\tinitialize program: /usr/bin/perl
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
14642:\ttransferring control: /usr/bin/perl
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
14642:\tcalling init:
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/DBI/DBI.so
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\tfind
library=libclntsh.so.10.1 [0]; searching
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
search path=
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] \t\t(RPATH from
file /usr/bin/perl)
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying
file=/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
search cache=/etc/ld.so.cache
*[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/local/instantclient_10_2/libclntsh.so.10.1*
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
search path=
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] /lib/tls/i686/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/lib/tls/i686
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/lib/tls/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/lib/tls
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/lib/i686/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/lib/i686
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/lib/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/lib
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
:/usr/lib/tls/i686/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/usr/lib/tls/i686
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/usr/lib/tls/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/usr/lib/tls
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/usr/lib/i686/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/usr/lib/i686
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/usr/lib/sse2
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] :/usr/lib
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] \t\t(system search
path)
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/lib/tls/i686/sse2/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/lib/tls/i686/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/lib/tls/sse2/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/lib/tls/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/lib/i686/sse2/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/lib/i686/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/lib/sse2/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/lib/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/tls/i686/sse2/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/tls/i686/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/tls/sse2/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/tls/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/i686/sse2/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/i686/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/sse2/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
trying file=/usr/lib/libclntsh.so.10.1
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
*[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] Uncaught
exception from user code:
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
\tinstall_driver(Oracle) failed: Can't load
'/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/D
BD/Oracle/Oracle.so' for module DBD::Oracle: libclntsh.so.10.1: cannot
open shared object file: No such file or directory at
/usr/lib/perl5/5.8.5/i386-linux-thr
ead-multi/DynaLoader.pm line 230.
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] at (eval 2) line 3
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] Compilation failed
in require at (eval 2) line 3.
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] Perhaps a required
shared library or dll isn't installed where expected
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] at
/var/www/cgi-bin/test2.pl line 27
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] at
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/DBI.pm line 715
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
\tDBI::install_driver('DBI', 'Oracle') called at
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/
DBI.pm line 565
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
\tDBI::connect('DBI', 'DBI:Oracle:epgy', 'bsears', 'bsears2change',
'HASH(0x9742c20)') called at /var/www/
cgi-bin/test2.pl line 27*
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1]
14642:\tcalling fini:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so [0]
[Thu Aug 30 14:33:14 2007] [error] [client 127.0.0.1] 14642:\t
.
.
.
the last lines in red show the dreaded "install_driver(Oracle) failed:
Can't load" error, but further up, you can see in the first highlighted
line that libclntsh.so.10.1 was actually searched for at the location it
exists (and where it was found when the script was run from the command
line!), but the search continued as if it were not found! so what the
heck is going on here? i could understand if the file were corrupted
(though we know it is not from the command line test) or possibly
unusable for some reason, but shouldn't the error message be something
other than "not found" at least? anyone?
thanks for bearing with this. it's too much detail, but perhaps not
enough (sigh).
bruce
