KARA VAN HORN created ZOOKEEPER-2200:
----------------------------------------
Summary: Perl ZooKeeper locks up during heavy load
Key: ZOOKEEPER-2200
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2200
Project: ZooKeeper
Issue Type: Bug
Components: c client
Affects Versions: 3.4.6
Environment: CentOS 5.8
Reporter: KARA VAN HORN
We are using Perl Net::ZooKeeper (0.38) and Net::ZooKeeper::Lock (0.03)
libraries. Deadlock appears to occur at the end during lock cleanup activity.
Here is a stack dump (sensitive names changed):
Thread 2 (Thread 0x2ac6fbfa3940 (LWP 13292)):
#0 0x00002ac6f5aed654 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00002ac6f5aeb47b in pthread_cond_signal@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#2 0x00002ac6f835539c in _zk_watcher (handle=<value optimized out>, type=2,
state=3, path=<value optimized out>, context=0x33f3ce0) at ZooKeeper.xs:179
#3 0x00002ac6f856d942 in do_foreach_watcher (zh=0x33e4fb0, type=2, state=3,
path=0x33f3f50 "/lock/cmts/cisco_device1.net-0001851215", list=0x33ed290)
at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zk_hashtable.c:279
#4 deliverWatchers (zh=0x33e4fb0, type=2, state=3, path=0x33f3f50
"/lock/cmts/cisco_device1.net-0001851215", list=0x33ed290)
at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zk_hashtable.c:321
#5 0x00002ac6f8564966 in process_completions (zh=0x33e4fb0) at
/home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zookeeper.c:2114
#6 0x00002ac6f856e101 in do_completion (v=<value optimized out>) at
/home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/mt_adaptor.c:466
#7 0x00002ac6f5ae683d in start_thread (arg=<value optimized out>) at
pthread_create.c:301
#8 0x00002ac6f5dd1fcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2ac6f6056af0 (LWP 12972)):
#0 0x00002ac6f5ae7c65 in pthread_join (threadid=47034119371072,
thread_return=0x0) at pthread_join.c:89
#1 0x00002ac6f856e7de in adaptor_finish (zh=0x33e4fb0) at
/home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/mt_adaptor.c:293
#2 0x00002ac6f8566cdc in zookeeper_close (zh=0x33e4fb0) at
/home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zookeeper.c:2536
#3 0x00002ac6f8357222 in XS_Net__ZooKeeper_DESTROY (my_perl=0x20df010,
cv=<value optimized out>) at ZooKeeper.xs:885
#4 0x00002ac6f4b38af6 in Perl_pp_entersub () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#5 0x00002ac6f4adb8d7 in ?? () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#6 0x00002ac6f4adf720 in Perl_call_sv () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#7 0x00002ac6f4b3d3c6 in Perl_sv_clear () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#8 0x00002ac6f4b3db70 in Perl_sv_free () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#9 0x00002ac6f4b6025c in Perl_free_tmps () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#10 0x00002ac6f4adf78a in Perl_call_sv () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#11 0x00002ac6f4b3d3c6 in Perl_sv_clear () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#12 0x00002ac6f4b3db70 in Perl_sv_free () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#13 0x00002ac6f4b3b0e5 in ?? () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#14 0x00002ac6f4b3b141 in Perl_sv_clean_objs () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#15 0x00002ac6f4ae185e in perl_destruct () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#16 0x0000000000401773 in main ()
There are about 4 out of 10,000 processes that end up in deadlock, and
according to our web searches, the only reason pthread_cond_signal would lock
is due to it waiting on an already destroyed condition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
