Hi T0m, Thanks, that's a very detailed email with a lot of ideas. It'll take a while to go through all of them.
You mentioned gdb. We've got a crontab which releases the locked process and one of the first things we did was to make gdb do a dump before killing the process. However, we haven't been able to interpret the output properly. I've attached an example below. It appears to be preparing an SQL statement but it doesn't give us much idea about which one. Have you (or anyone else) got any idea what all this means? not ok Sat Jan 8 18:23:01 2011: http://www.xxxx.co.uk/use_ws_1 500 read timeout 11842 perl /usr/local/etc/httpd/fcgi/catalyst/xxxx/script/xxxx_fastcgi.pl not ok Sat Jan 8 18:23:01 2011: http://www.xxxx.co.uk/use_ws_2 500 read timeout GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.1) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Attaching to process 11842 Reading symbols from /usr/bin/perl...(no debugging symbols found)...done. Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libutil.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Util/Util.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Util/Util.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Sub/Name/Name.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Sub/Name/Name.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Devel/GlobalDestruction/GlobalDestruction.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Devel/GlobalDestruction/GlobalDestruction.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/MOP/MOP.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/MOP/MOP.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Moose/Moose.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Moose/Moose.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Variable/Magic/Magic.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Variable/Magic/Magic.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Sub/Identify/Identify.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Sub/Identify/Identify.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Validate/Validate.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Validate/Validate.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/MooseX/Role/WithOverloading/WithOverloading.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/MooseX/Role/WithOverloading/WithOverloading.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Encode.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Encode.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Cache/FastMmap/CImpl/CImpl.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Cache/FastMmap/CImpl/CImpl.so Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/FCGI/FCGI.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/FCGI/FCGI.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/XSAccessor/XSAccessor.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Class/XSAccessor/XSAccessor.so Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Date/Calc/Calc.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Date/Calc/Calc.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/PerlIO/encoding/encoding.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/PerlIO/encoding/encoding.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBI/DBI.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBI/DBI.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/Oracle/Oracle.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/Oracle/Oracle.so Reading symbols from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1...(no debugging symbols found)...done. Loaded symbols for /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 Reading symbols from /oracle/product/10.1.0/db_1/lib/libnnz10.so...(no debugging symbols found)...done. Loaded symbols for /oracle/product/10.1.0/db_1/lib/libnnz10.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so Reading symbols from /usr/local/lib/libexpat.so.0...done. Loaded symbols for /usr/local/lib/libexpat.so.0 Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/DNS/DNS.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Math/BigInt/FastCalc/FastCalc.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Math/BigInt/FastCalc/FastCalc.so Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Image/Magick/Magick.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Image/Magick/Magick.so Reading symbols from /usr/lib64/libMagickCore.so.1...done. Loaded symbols for /usr/lib64/libMagickCore.so.1 Reading symbols from /usr/lib64/libMagickWand.so.1...done. Loaded symbols for /usr/lib64/libMagickWand.so.1 Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libfreetype.so.6 Reading symbols from /usr/lib64/libtiff.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libtiff.so.3 Reading symbols from /usr/lib64/libjpeg.so.62...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libjpeg.so.62 Reading symbols from /usr/lib64/libfontconfig.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libfontconfig.so.1 Reading symbols from /usr/lib64/libXext.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libXext.so.6 Reading symbols from /usr/lib64/libSM.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libSM.so.6 Reading symbols from /usr/lib64/libICE.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libICE.so.6 Reading symbols from /usr/lib64/libX11.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libX11.so.6 Reading symbols from /usr/lib64/libXt.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libXt.so.6 Reading symbols from /usr/lib64/libbz2.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libbz2.so.1 Reading symbols from /usr/local/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from /usr/lib64/libXau.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libXau.so.6 Reading symbols from /usr/lib64/libXdmcp.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libXdmcp.so.6 Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/SDBM_File/SDBM_File.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/SDBM_File/SDBM_File.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/String/String.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/String/String.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/Map/Map.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/Map/Map.so Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/Map8/Map8.so...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Unicode/Map8/Map8.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Unicode/Unicode.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Unicode/Unicode.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/JP/JP.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/JP/JP.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/B/B.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/B/B.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DateTime/DateTime.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/DateTime/DateTime.so Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Clone/Clone.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Clone/Clone.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Text/CSV_XS/CSV_XS.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Text/CSV_XS/CSV_XS.so Reading symbols from /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Template/Stash/XS/XS.so...done. Loaded symbols for /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Template/Stash/XS/XS.so Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Digest/SHA1/SHA1.so Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Crypt/SSLeay/SSLeay.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Crypt/SSLeay/SSLeay.so 0x0000003bbe20d8e0 in __read_nocancel () from /lib64/libpthread.so.0 (gdb) (gdb) #0 0x0000003bbe20d8e0 in __read_nocancel () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00002ac7089512a2 in snttread () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #2 0x00002ac70894e42a in ?? () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #3 0x00002ac708818d42 in nsprecv () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #4 0x00002ac70881c8e3 in nsrdr () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #5 0x00002ac7087f6cc8 in nsdo () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #6 0x00002ac7087f662a in nsbrecv () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #7 0x00002ac708831c0d in nioqrc () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #8 0x00002ac708992985 in ttcdrv () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #9 0x00002ac70883a4fc in nioqwa () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #10 0x00002ac708657c4f in upirtrc () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #11 0x00002ac7085c1193 in kpurcsc () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #12 0x00002ac70856fbcf in kpuexecv8 () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #13 0x00002ac70856d7b3 in kpuexec () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #14 0x00002ac70861bb27 in OCIStmtExecute () from /oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 No symbol table info available. #15 0x00002ac70803adc9 in ora_describe (h=0x108af110, imp_sth=0x15fdfa10) at oci8.c:1997 my_perl = <value optimized out> imp_dbh = 0x15594520 long_readlen = 6553500 num_fields = <value optimized out> num_errors = <value optimized out> has_longs = <value optimized out> est_width = <value optimized out> nested_cursors = <value optimized out> i = <value optimized out> status = <value optimized out> #16 0x00002ac70803e274 in ora_st_prepare (sth=0x108af110, imp_sth=0x15fdfa10, statement=<value optimized out>, attribs=<value optimized out>) at oci8.c:337 my_perl = <value optimized out> imp_dbh = <value optimized out> status = 0 oparse_lng = 1 ora_check_sql = 1 ora_placeholders = 48 ora_pad_empty = 0x15519200 #17 0x00002ac7080215c1 in XS_DBD__Oracle__st__prepare (my_perl=0xd8b8010, cv=<value optimized out>) at Oracle.xsi:434 imp_sth = 0x15fdfa10 sth = 0x108af110 statement = 0x107204b0 attribs = 0x0 sp = <value optimized out> ax = <value optimized out> items = <value optimized out> #18 0x0000003bc2e90a96 in Perl_pp_entersub () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #19 0x0000003bc2e8a33e in Perl_runops_standard () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #20 0x0000003bc2e37576 in Perl_call_sv () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #21 0x00002ac707de97da in XS_DBI_dispatch (my_perl=0xd8b8010, cv=0xef1e7e0) at DBI.xs:3459 sp = 0x0 ax = 6 items = 2 perinterp_sv = <value optimized out> PERINTERP = 0x10740aa0 h = 0xff2bdc0 st1 = 0x10742000 st2 = 0xef1e7e0 err_sv = <value optimized out> tmp_svp = <value optimized out> hook_svp = 0xc0d8b8010 mg = <value optimized out> gimme = 2064 trace_flags = 0 trace_level = 0 is_DESTROY = 0 is_unrelated_to_Statement = 0 keep_error = 0 ErrCount = 18446744073709551615 i = 227246096 outitems = 228021872 call_depth = 1 is_nested_call = 0 profile_t1 = 0 meth_name = 0x10865d70 "prepare" ima = 0x10865c00 ima_flags = <value optimized out> imp_xxh = 0x15594520 imp_msv = 0x15a64c36 qsv = 0x58 #22 0x0000003bc2e90a96 in Perl_pp_entersub () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #23 0x0000003bc2e8a33e in Perl_runops_standard () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #24 0x0000003bc2e37e9c in perl_run () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #25 0x00000000004017bc in main () No symbol table info available. (gdb) A debugging session is active. Inferior 1 [process 11842] will be detached. Quit anyway? (y or n) [answered Y; input not from terminal] Detaching from program: /usr/bin/perl, process 11842 kill process 11842 Regards Duncan -----Original Message----- From: Tomas Doran [mailto:[email protected]] Sent: 11 January 2011 11:16 To: The elegant MVC web framework Subject: Re: [Catalyst] Zeus and Catalyst On 10 Jan 2011, at 22:05, Duncan Garland wrote: > Are there any known problems using two serving running Zeus with a > load balancer, fastcgi, Catalyst and Oracle 10.2 on Red Hat? None known :) > Our servers lock up occasionally. Half a dozen times a day, which is > often enough to cause real embarassment. We restart them by killing > the fastcgi process. > > The symptom is that something causes Catalyst to start returning > zero length pages. > > eg > > 77.242.199.1 - - [10/Jan/2011:20:22:22 +0000] "GET /fcgi/catalyst/ > xxxx/script/xxxx_fastcgi.pl/javascript/calculation_accept HTTP/1.1" > 200 0 "http://www.xxxx.co.uk/fcgi/catalyst/xxxx/script/xxxx_fastcgi.pl/home > " "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.13) > Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729)" > > Half a dozen calls return this, then the server locks up completely > until the process is killed. > Can you define 'locks up completely' here - is it spinning at 100% CPU? What does strace say the fcgi process is doing when it's locked up? Also, what does gdb say the stack is? Is this fcgi on a unix domain socket, or a tcp socket? > It doesn't happen on our other servers which don't have load > balancers. One of the first things we did was to force the load > balancer to always use the same server. That didn't help. What is the balancer doing for it's health checks? Maybe this is what is making it sick for some reason? (I've seen cases (non-Catalyst) where load balancers making half requests / slightly mad requests / closing the socket before getting a response / other mad things have caused massive lossage, so it's worth thinking about) > Could it be jumping out of the error handling and actually returning > zero content? I can't see where. It could be crapping itself half way through error handling? > Is there a module dependency which would cause this? No, I don't think so, but try running the latest release of FCGI.pm to be sure.. > Is it Zeus or fastcgi? One of these - something I _have_ seen in the past is that some (oh hai nginx!) web server's fcgi implementations can get real confused if you output a load of stuff to the fcgi error channel.. So try running your fastcgi with --keeperr, see if that has any effect.. Also, try loading Devel::SimpleTrace - with --keeperr and ::SimpleTrace then you're more likely to get helpful errors out before it all locks up. Cheers t0m P.S. Once you have --keeperr on, then trying this: http://use.perl.org/~jjore/journal/39319 could be useful to get a perl level backtrace out.. _______________________________________________ List: [email protected] Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[email protected]/ Dev site: http://dev.catalyst.perl.org/ _______________________________________________ List: [email protected] Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[email protected]/ Dev site: http://dev.catalyst.perl.org/
