Ok. The log shows that the connection is being closed. So it just looks like a network issue causing the connection to be unstable.
Looks like a racing condition to me if this theory is correct. You may need to review the object and add a mutex around the critical sections. On Tue, Mar 3, 2020 at 10:07 AM Xiaoyao Qian <[email protected]> wrote: > During normal running state. And it happens only to one data center.. > > On Tue, Mar 3, 2020 at 9:33 AM Ning Wang <[email protected]> wrote: > >> Yeah. Segfault is more likely in this case. But maybe not for the object >> itself but for the vtable pointer. >> >> Pure function call might be possible though I think. It all depends on >> the 4 bytes in the address. It could happen If a new object of a different >> type has been created in the place or an object is moved to the memory >> block. >> >> More investigation is necessary. >> >> When does it happen? normal running state or when topo or instance is >> shutting down? >> >> >> On Tue, Mar 3, 2020 at 8:34 AM Xiaoyao Qian <[email protected]> wrote: >> >>> I agreed with the second possibility, but wouldn’t it cause segfault if >>> the object has been deleted but the method is still invoked? >>> >>> On Tue, Mar 3, 2020 at 01:16 Ning Wang <[email protected]> wrote: >>> >>>> Looks like the exception is caused by pure virtual function calling. >>>> >>>> Both exceptions are from BaseClient. It seems like the Client object >>>> doesn't have the virtual functions implemented, which is not expected. >>>> >>>> Another possibility is that the client object has been deleted hence >>>> the vtable is not valid any more. This could be something you can check >>>> given the last log shows "Stmgr stmgr-12 closed connection". >>>> >>>> >>>> On Mon, Mar 2, 2020 at 12:20 PM Huijun Wu <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> We observed the Stmgr coredump, see the below logs. Anybody has any >>>>> idea on >>>>> this? Thanks. >>>>> >>>>> Best, >>>>> Huijun >>>>> >>>>> ------------ >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: pure virtual >>>>> method >>>>> called >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: terminate called >>>>> without an active exception >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: *** Aborted at >>>>> 1582847383 (unix time) try "date -d @1582847383" if you are using GNU >>>>> date >>>>> *** >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: PC: @ >>>>> 0x7f5bc42da277 __GI_raise >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: *** SIGABRT >>>>> (@0xbcb0000b338) received by PID 45880 (TID 0x7f5bc54c1780) from PID >>>>> 45880; >>>>> stack trace: *** >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7f5bc50a76d0 (unknown) >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7f5bc42da277 __GI_raise >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7f5bc42db968 __GI_abort >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7f5bc48e77d5 __gnu_cxx::__verbose_terminate_handler() >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7f5bc48e5746 (unknown) >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7f5bc48e5773 std::terminate() >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7f5bc48e62df __cxa_pure_virtual >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4dfe1d BaseClient::OnConnect() >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4de404 EventLoopImpl::handleWriteCallback() >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4f0a9f event_process_active_single_queue >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4f115f event_base_loop >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x40c81f main >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7f5bc42c6445 __libc_start_main >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x41257c (unknown) >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x0 (unknown) >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: >>>>> ------------ >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: pure virtual >>>>> method >>>>> called >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: terminate called >>>>> without an active exception >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: *** Aborted at >>>>> 1582837876 (unix time) try "date -d @1582837876" if you are using GNU >>>>> date >>>>> *** >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: PC: @ >>>>> 0x7fe9d8935277 __GI_raise >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: *** SIGABRT >>>>> (@0xbcb0005176a) received by PID 333674 (TID 0x7fe9d9b1c780) from PID >>>>> 333674; stack trace: *** >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7fe9d97026d0 (unknown) >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7fe9d8935277 __GI_raise >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7fe9d8936968 __GI_abort >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7fe9d8f427d5 __gnu_cxx::__verbose_terminate_handler() >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7fe9d8f40746 (unknown) >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7fe9d8f40773 std::terminate() >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7fe9d8f412df __cxa_pure_virtual >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4df98a BaseClient::Start_Base() >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x443dfa heron::stmgr::StMgrClient::OnReConnectTimer() >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4df6ed >>>>> >>>>> _ZNSt17_Function_handlerIFvN9EventLoop6StatusEEZN10BaseClient13AddTimer_BaseESt8functionIFvvEElEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4ded81 EventLoopImpl::handleTimerCallback() >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4dc840 EventLoopImpl::eventLoopImplTimerCallback() >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4f0a9f event_process_active_single_queue >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x4f115f event_base_loop >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x40c81f main >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x7fe9d8921445 __libc_start_main >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x41257c (unknown) >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>>> 0x0 (unknown) >>>>> [2020-02-27 21:11:17 +0000] [INFO]: stmgr-233 stdout: >>>>> -------------------- >>>>> I0227 21:11:16.813194 333674 stmgr-client.cpp:139] Will try to >>>>> reconnect >>>>> again after 1 seconds >>>>> E0227 21:11:16.813606 333674 baseconnection.cpp:142] BufferEvent >>>>> reported >>>>> error on connection 0x2d2b720 >>>>> I0227 21:11:16.813617 333674 stmgr-client.cpp:133] Stmgr stmgr-154 >>>>> running >>>>> at xxxxxxx:31831 closed connection with code Write error >>>>> I0227 21:11:16.813670 333674 stmgr-client.cpp:139] Will try to >>>>> reconnect >>>>> again after 1 seconds >>>>> E0227 21:11:16.814357 333674 baseconnection.cpp:142] BufferEvent >>>>> reported >>>>> error on connection 0x2d3aa00 >>>>> I0227 21:11:16.814375 333674 stmgr-server.cpp:111] StMgrServer Got >>>>> connection close of 0x2d3aa00 from yyyyyyyy:35270 >>>>> I0227 21:11:16.814378 333674 stmgr-server.cpp:121] Stmgr stmgr-86 >>>>> closed >>>>> connection >>>>> E0227 21:11:16.814669 333674 baseconnection.cpp:142] BufferEvent >>>>> reported >>>>> error on connection 0x2d3a220 >>>>> I0227 21:11:16.814683 333674 stmgr-server.cpp:111] StMgrServer Got >>>>> connection close of 0x2d3a220 from yyyyyyyy:33974 >>>>> I0227 21:11:16.814685 333674 stmgr-server.cpp:121] Stmgr stmgr-154 >>>>> closed >>>>> connection >>>>> E0227 21:11:16.816232 333674 baseconnection.cpp:142] BufferEvent >>>>> reported >>>>> error on connection 0x56ab800 >>>>> I0227 21:11:16.816256 333674 stmgr-client.cpp:133] Stmgr stmgr-221 >>>>> running >>>>> at xxxxxxx:31294 closed connection with code Write error >>>>> I0227 21:11:16.816263 333674 stmgr-client.cpp:139] Will try to >>>>> reconnect >>>>> again after 1 seconds >>>>> E0227 21:11:16.817833 333674 baseconnection.cpp:142] BufferEvent >>>>> reported >>>>> error on connection 0x2d295e0 >>>>> I0227 21:11:16.817849 333674 stmgr-client.cpp:133] Stmgr stmgr-12 >>>>> running >>>>> at xxxxxxx:31846 closed connection with code Write error >>>>> I0227 21:11:16.817853 333674 stmgr-client.cpp:139] Will try to >>>>> reconnect >>>>> again after 1 seconds >>>>> E0227 21:11:16.820160 333674 baseconnection.cpp:142] BufferEvent >>>>> reported >>>>> error on connection 0x2d39500 >>>>> I0227 21:11:16.820192 333674 stmgr-server.cpp:111] StMgrServer Got >>>>> connection close of 0x2d39500 from yyyyyyyy:41220 >>>>> I0227 21:11:16.820196 333674 stmgr-server.cpp:121] Stmgr stmgr-12 >>>>> closed >>>>> connection >>>>> >>>> -- >>> Thanks >>> Xiaoyao >>> >> > > -- > Thanks > Xiaoyao >
