To follow up, with the hackaround in the previous post, and modify
test.pl to dispatch batches of 20 children, it eventually dies with a
segfault (below - coredump available on request). Although I can not be
sure if this is related to my hack, or the core library.
Thanks,
Justin
Message: Process 1141691 (test.pl) of user 1000 dumped core.
Module libnss_resolve.so.2 from rpm
systemd-253.10-1.fc38.x86_64
Module libnss_mdns4_minimal.so.2 from rpm
nss-mdns-0.15.1-8.fc38.x86_64
Module libcap.so.2 from rpm libcap-2.48-6.fc38.x86_64
Module libnss_myhostname.so.2 from rpm
systemd-253.10-1.fc38.x86_64
Module libnuma.so.1 from rpm numactl-2.0.16-2.fc38.x86_64
Module libaio.so.1 from rpm libaio-0.3.111-15.fc38.x86_64
Module libcrypt.so.2 from rpm
libxcrypt-4.4.36-1.fc38.x86_64
Stack trace of thread 1141697:
#0 0x00007f8bfe0b0884 __pthread_kill_implementation
(libc.so.6 + 0x8e884)
#1 0x00007f8bfe05fafe raise (libc.so.6 + 0x3dafe)
#2 0x00007f8beeccb73f skgesigOSCrash
(libclntsh.so.21.1 + 0x28cb73f)
#3 0x00007f8bef3ca9ed kpeDbgSignalHandler
(libclntsh.so.21.1 + 0x2fca9ed)
#4 0x00007f8beeccba52 skgesig_sigactionHandler
(libclntsh.so.21.1 + 0x28cba52)
#5 0x00007f8bfe05fbb0 __restore_rt (libc.so.6 + 0x3dbb0)
#6 0x00007f8bfe302e70 Perl_csighandler3
(libperl.so.5.36 + 0x102e70)
#7 0x00007f8bfe05fbb0 __restore_rt (libc.so.6 + 0x3dbb0)
#8 0x00007f8bfe0ab219 __futex_abstimed_wait_common
(libc.so.6 + 0x89219)
#9 0x00007f8bfe0adf22
pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x8bf22)
#10 0x00007f8beb6abfc3 sltspctimewait
(libclntshcore.so.21.1 + 0xabfc3)
#11 0x00007f8bec8cf501 kpucpincrtime (libclntsh.so.21.1
+ 0x4cf501)
#12 0x00007f8bfe0ae947 start_thread (libc.so.6 + 0x8c947)
#13 0x00007f8bfe134860 __clone3 (libc.so.6 + 0x112860)
Stack trace of thread 1141691:
#0 0x00007f8bfe1230ea read (libc.so.6 + 0x1010ea)
#1 0x00007f8bef8d02d0 snttread (libclntsh.so.21.1 +
0x34d02d0)
#2 0x00007f8bef8ceb35 nttfprd (libclntsh.so.21.1 +
0x34ceb35)
#3 0x00007f8bef8c3c95 nsbasic_brc (libclntsh.so.21.1 +
0x34c3c95)
#4 0x00007f8bef8b7e7f nioqrc (libclntsh.so.21.1 +
0x34b7e7f)
#5 0x00007f8bef8f7ecf ttcdrv (libclntsh.so.21.1 +
0x34f7ecf)
#6 0x00007f8bef8bdccd nioqwa (libclntsh.so.21.1 +
0x34bdccd)
#7 0x00007f8bef88b690 upirtrc (libclntsh.so.21.1 +
0x348b690)
#8 0x00007f8bef8a0331 kpurcsc (libclntsh.so.21.1 +
0x34a0331)
#9 0x00007f8bef890f40 kpuexec (libclntsh.so.21.1 +
0x3490f40)
#10 0x00007f8bef88ad49 OCIStmtExecute
(libclntsh.so.21.1 + 0x348ad49)
#11 0x00007f8bfe5e2377 ora_st_execute (Oracle.so + 0x24377)
#12 0x00007f8bfe5cad7d XS_DBD__Oracle__st_execute
(Oracle.so + 0xcd7d)
#13 0x00007f8bfe61b106 XS_DBI_dispatch (DBI.so + 0xf106)
#14 0x00007f8bfe3265aa Perl_pp_entersub
(libperl.so.5.36 + 0x1265aa)
#15 0x00007f8bfe317958 Perl_runops_standard
(libperl.so.5.36 + 0x117958)
#16 0x00007f8bfe27c3e7 Perl_call_sv (libperl.so.5.36 +
0x7c3e7)
#17 0x00007f8bfe61b164 XS_DBI_dispatch (DBI.so + 0xf164)
#18 0x00007f8bfe3265aa Perl_pp_entersub
(libperl.so.5.36 + 0x1265aa)
#19 0x00007f8bfe317958 Perl_runops_standard
(libperl.so.5.36 + 0x117958)
#20 0x00007f8bfe28259d perl_run (libperl.so.5.36 + 0x8259d)
#21 0x000056551ca1734a main (perl + 0x134a)
#22 0x00007f8bfe049b8a __libc_start_call_main
(libc.so.6 + 0x27b8a)
#23 0x00007f8bfe049c4b __libc_start_main@@GLIBC_2.34
(libc.so.6 + 0x27c4b)
#24 0x000056551ca17385 _start (perl + 0x1385)
ELF object binary architecture: AMD x86-64
On 2023/10/25 15:39, Justin Schoeman wrote:
Good day,
I am trying to dig into an issue after a recent Oracle update. The
application has been running flawlessly from version 9.x to 18.5, and
has only broken on the most recent client library updates.
Attached is a cut down application which demonstrates the issue.
The forked child processes never exit. Attaching to them with GDB gives:
(gdb) t a a bt
Thread 1 (Thread 0x7f0c48a62c40 (LWP 773676) "test.pl"):
#0 futex_wait (private=<optimized out>, expected=14,
futex_word=0x55c5ef1c96e4) at ../sysdeps/nptl/futex-internal.h:146
#1 futex_wait_simple (private=<optimized out>, expected=14,
futex_word=0x55c5ef1c96e4) at ../sysdeps/nptl/futex-internal.h:177
#2 __GI___pthread_cond_destroy (cond=0x55c5ef1c96c0) at
pthread_cond_destroy.c:53
#3 0x00007f0c35cac072 in sltspcdestroy () from
/usr/lib/oracle/21/client64/lib/libclntshcore.so.21.1
#4 0x00007f0c36ecf68e in kpucpstopthr () from
/usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
#5 0x00007f0c39e9a42e in kpufhndl0 () from
/usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
#6 0x00007f0c48a1028a in ora_db_destroy
(dbh=dbh@entry=0x55c5ef32c5b0, imp_dbh=imp_dbh@entry=0x55c5ef035390)
at /root/.local/share/.cpan/build/DBD-Oracle-1.83-2/dbdimp.c:1263
#7 0x00007f0c48a10752 in XS_DBD__Oracle__db_DESTROY
(my_perl=0x55c5eee272a0, cv=<optimized out>) at ./Oracle.xsi:432
#8 0x00007f0c48a4d106 in XS_DBI_dispatch (my_perl=0x55c5eee272a0,
cv=0x55c5ef13b9c8) at
/usr/src/debug/perl-DBI-1.643-15.fc38.x86_64/DBI.xs:3783
#9 0x00007f0c487265aa in Perl_pp_entersub (my_perl=0x55c5eee272a0) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/pp_hot.c:5353
#10 0x00007f0c4867c7a6 in Perl_call_sv (my_perl=<optimized out>,
sv=<optimized out>, flags=<optimized out>) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/perl.c:3057
#11 0x00007f0c4872c643 in S_curse
(my_perl=my_perl@entry=0x55c5eee272a0, sv=sv@entry=0x55c5ef34af38,
check_refcnt=check_refcnt@entry=true) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/sv.c:6973
#12 0x00007f0c4872cd88 in Perl_sv_clear
(my_perl=my_perl@entry=0x55c5eee272a0,
orig_sv=orig_sv@entry=0x55c5ef079858) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/sv.c:6536
#13 0x00007f0c4872d3a2 in Perl_sv_free2 (my_perl=0x55c5eee272a0,
sv=0x55c5ef079858, rc=<optimized out>) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/sv.c:7073
#14 0x00007f0c486fc878 in Perl_SvREFCNT_dec (sv=<optimized out>,
my_perl=0x55c5eee272a0) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/inline.h:405
#15 S_mg_free_struct (my_perl=0x55c5eee272a0, sv=<optimized out>,
mg=0x55c5ef34bb70) at /usr/src/debug/perl-5.36.1-497.fc38.x86_64/mg.c:563
#16 0x00007f0c486fc8c9 in Perl_mg_free (my_perl=0x55c5eee272a0,
sv=0x55c5ef34aec0) at /usr/src/debug/perl-5.36.1-497.fc38.x86_64/mg.c:585
#17 0x00007f0c4872cdc3 in Perl_sv_clear
(my_perl=my_perl@entry=0x55c5eee272a0,
orig_sv=orig_sv@entry=0x55c5ef34aec0) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/sv.c:6544
#18 0x00007f0c4872d3a2 in Perl_sv_free2 (my_perl=0x55c5eee272a0,
sv=0x55c5ef34aec0, rc=<optimized out>) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/sv.c:7073
#19 0x00007f0c48722646 in Perl_SvREFCNT_dec_NN (sv=<optimized out>,
my_perl=<optimized out>) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/inline.h:419
#20 do_clean_objs (ref=0x55c5ef338ce0, my_perl=<optimized out>) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/sv.c:448
#21 S_visit (mask=2048, flags=2048, f=<optimized out>,
my_perl=0x55c5eee272a0) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/sv.c:391
#22 Perl_sv_clean_objs (my_perl=0x55c5eee272a0) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/sv.c:542
#23 0x00007f0c48680160 in perl_destruct (my_perl=0x55c5eee272a0) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/perl.c:880
#24 0x000055c5eda0f324 in main (argc=<optimized out>, argv=<optimized
out>, env=<optimized out>) at
/usr/src/debug/perl-5.36.1-497.fc38.x86_64/perlmain.c:121
I have instrumented dbdimpl.c as follows:
/* free global environment handle during
destruction of last connection */
} else if ( (imp_dbh->envhp == imp_drh->envhp)
&& (SvTRUE(perl_get_sv("DBI::PERL_ENDING",0))) ) {
PerlIO_printf(PerlIO_stderr(),
"EXIT1\n");
OCIHandleFree_log_stat(imp_dbh,
imp_dbh->envhp, OCI_HTYPE_ENV, status);
PerlIO_printf(PerlIO_stderr(),
"EXIT2\n");
if ( status == OCI_SUCCESS ) {
PerlIO_printf(PerlIO_stderr(),
"EXIT3\n");
imp_dbh->envhp = NULL;
imp_drh->envhp = NULL;
}
}
And the child prints 'EXIT1', but never 'EXIT2'. If I comment out the
call to OCIHandleFree_log_stat() then the child process exits without
error (although there are random segfaults after long run times).
1. A complete log of all steps of the build, e.g.:
Atatched (build.log)
2. Full details of which version of Oracle client and server you're using
# rpm -qa | grep oracle
oracle-instantclient-basic-21.11.0.0.0-1.x86_64
oracle-instantclient-devel-21.11.0.0.0-1.x86_64
3. The output of perl -V (that's a capital V, not lowercase)
Attached (perl-V.log)
Thanks,
Justin
