git-hulk opened a new issue, #2512: URL: https://github.com/apache/kvrocks/issues/2512
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues. ### Version unstable ### Minimal reproduce step None ### What did you expect to see? Won't cause deadlock at any situation ### What did you see instead? The worker threads are stuck after switching master frequently: ``` (gdb) thread 8 [Switching to thread 8 (Thread 0x7fbeb4bf8640 (LWP 3166409))] #0 __futex_abstimed_wait_common (cancel=false, private=<optimized out>, abstime=0x0, clockid=0, expected=3, futex_word=0x7ffc2c7fe668) at ./nptl/futex-internal.c:103 103 in ./nptl/futex-internal.c (gdb) bt #0 __futex_abstimed_wait_common (cancel=false, private=<optimized out>, abstime=0x0, clockid=0, expected=3, futex_word=0x7ffc2c7fe668) at ./nptl/futex-internal.c:103 #1 __GI___futex_abstimed_wait64 (futex_word=futex_word@entry=0x7ffc2c7fe668, expected=expected@entry=3, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:128 #2 0x00007fbebeb227fb in __pthread_rwlock_rdlock_full64 (abstime=0x0, clockid=0, rwlock=0x7ffc2c7fe660) at ./nptl/pthread_rwlock_common.c:460 #3 ___pthread_rwlock_rdlock (rwlock=0x7ffc2c7fe660) at ./nptl/pthread_rwlock_rdlock.c:26 #4 0x0000561e231e2028 in std::__glibcxx_rwlock_rdlock (__rwlock=<optimized out>) at /usr/include/c++/11/shared_mutex:78 #5 std::__shared_mutex_pthread::lock_shared (this=<optimized out>) at /usr/include/c++/11/shared_mutex:229 #6 std::shared_mutex::lock_shared (this=<optimized out>) at /usr/include/c++/11/shared_mutex:426 #7 std::shared_lock<std::shared_mutex>::shared_lock (__m=..., this=<optimized out>) at /usr/include/c++/11/shared_mutex:727 #8 Server::WorkConcurrencyGuard (this=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/server.cc:637 #9 redis::Connection::ExecuteCommands (this=<optimized out>, to_process_cmds=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/redis_connection.cc:355 #10 0x0000561e230dc2ac in redis::Connection::OnRead (bev=<optimized out>, this=0x7fbeb23cb580) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/redis_connection.cc:92 #11 EvbufCallbackBase<redis::Connection, true, true, true>::readCB (bev=<optimized out>, ctx=0x7fbeb23cb580) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/common/event_util.h:70 #12 0x0000561e2383be12 in bufferevent_run_deferred_callbacks_unlocked (cb=<optimized out>, arg=0x7fbeb23cb300) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/build/_deps/libevent-src/bufferevent.c:208 #13 0x0000561e2384271d in event_process_active_single_queue (base=base@entry=0x7fbebe64c900, activeq=0x7fbebe651060, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/build/_deps/libevent-src/event.c:1720 #14 0x0000561e23843117 in event_process_active (base=0x7fbebe64c900) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/build/_deps/libevent-src/event.c:1783 #15 event_base_loop (base=0x7fbebe64c900, flags=0) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/build/_deps/libevent-src/event.c:2006 #16 0x0000561e231f5f3e in Worker::Run (tid=..., this=0x7fbebe6ab900) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/worker.cc:293 #17 operator() (__closure=<optimized out>, __closure=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/worker.cc:546 #18 operator() (__closure=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/common/thread_util.h:38 #19 std::__invoke_impl<void, util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > (__f=...) at /usr/include/c++/11/bits/invoke.h:61 #20 std::__invoke<util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > (__fn=...) at /usr/include/c++/11/bits/invoke.h:96 #21 std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > >::_M_invoke<0> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259 #22 std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > >::operator() ( this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266 #23 std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > > >::_M_run(void) (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211 #24 0x0000561e23a3ec34 in execute_native_thread_routine () #25 0x00007fbebeb1cac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #26 0x00007fbebebae850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 (gdb) thread 10 [Switching to thread 10 (Thread 0x7fbeb6ffe640 (LWP 3166411))] #0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=3180509, futex_word=0x7fbe574ff910) at ./nptl/futex-internal.c:57 57 in ./nptl/futex-internal.c (gdb) bt #0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=3180509, futex_word=0x7fbe574ff910) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=3180509, futex_word=0x7fbe574ff910) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fbe574ff910, expected=3180509, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139 #3 0x00007fbebeb1e624 in __pthread_clockjoin_ex (threadid=140455485371968, thread_return=0x0, clockid=0, abstime=0x0, block=<optimized out>) at ./nptl/pthread_join_common.c:105 #4 0x0000561e23a3eca7 in std::thread::join() () #5 0x0000561e2328c52e in util::ThreadOperationImpl<&std::thread::join>(std::thread&, char const*) [clone .constprop.0] (t=..., op=0x561e23a94184 "join") at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/common/thread_util.cc:39 #6 0x0000561e230c6bd4 in util::ThreadJoin (t=...) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/common/thread_util.cc:47 #7 ReplicationThread::Stop (this=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/cluster/replication.cc:367 #8 0x0000561e231e62bb in Server::AddMaster (this=0x7ffc2c7fe150, host=..., port=6379, force_reconnect=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/server.cc:261 #9 0x0000561e230bfa42 in Cluster::SetMasterSlaveRepl (this=0x7fbeba875600) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/cluster/cluster.cc:245 #10 0x0000561e230c0697 in Cluster::SetClusterNodes (this=0x7fbeba875600, nodes_str=..., version=<optimized out>, force=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/cluster/cluster.cc:205 #11 0x0000561e230ee13d in redis::CommandClusterX::Execute (this=0x7fbe3001e000, srv=0x7ffc2c7fe150, conn=<optimized out>, output=0x7fbeb6ff5450) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/commands/cmd_cluster.cc:224 #12 0x0000561e231e236f in redis::Connection::ExecuteCommands (this=<optimized out>, to_process_cmds=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/redis_connection.cc:426 #13 0x0000561e230dc2ac in redis::Connection::OnRead (bev=<optimized out>, this=0x7fbe16db5680) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/redis_connection.cc:92 #14 EvbufCallbackBase<redis::Connection, true, true, true>::readCB (bev=<optimized out>, ctx=0x7fbe16db5680) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/common/event_util.h:70 #15 0x0000561e2383be12 in bufferevent_run_deferred_callbacks_unlocked (cb=<optimized out>, arg=0x7fbe16db5400) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/build/_deps/libevent-src/bufferevent.c:208 #16 0x0000561e2384271d in event_process_active_single_queue (base=base@entry=0x7fbebe64cf00, activeq=0x7fbebe6512c0, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/build/_deps/libevent-src/event.c:1720 #17 0x0000561e23843117 in event_process_active (base=0x7fbebe64cf00) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/build/_deps/libevent-src/event.c:1783 #18 event_base_loop (base=0x7fbebe64cf00, flags=0) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/build/_deps/libevent-src/event.c:2006 #19 0x0000561e231f5f3e in Worker::Run (tid=..., this=0x7fbebe6abb00) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/worker.cc:293 #20 operator() (__closure=<optimized out>, __closure=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/server/worker.cc:546 #21 operator() (__closure=<optimized out>) at /root/kvrocks-zip-build-cd/kvrocks-zip-build/kvrocks/src/common/thread_util.h:38 #22 std::__invoke_impl<void, util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > (__f=...) at /usr/include/c++/11/bits/invoke.h:61 #23 std::__invoke<util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > (__fn=...) at /usr/include/c++/11/bits/invoke.h:96 #24 std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > >::_M_invoke<0> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259 #25 std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > >::operator() ( this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266 #26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::<lambda()> >(char const*, WorkerThread::Start()::<lambda()>)::<lambda()> > > >::_M_run(void) (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211 #27 0x0000561e23a3ec34 in execute_native_thread_routine () #28 0x00007fbebeb1cac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #29 0x00007fbebebae850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 (gdb) ``` ### Anything Else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
