During normal running state. And it happens only to one data center.. On Tue, Mar 3, 2020 at 9:33 AM Ning Wang <[email protected]> wrote:
> Yeah. Segfault is more likely in this case. But maybe not for the object > itself but for the vtable pointer. > > Pure function call might be possible though I think. It all depends on the > 4 bytes in the address. It could happen If a new object of a different type > has been created in the place or an object is moved to the memory block. > > More investigation is necessary. > > When does it happen? normal running state or when topo or instance is > shutting down? > > > On Tue, Mar 3, 2020 at 8:34 AM Xiaoyao Qian <[email protected]> wrote: > >> I agreed with the second possibility, but wouldn’t it cause segfault if >> the object has been deleted but the method is still invoked? >> >> On Tue, Mar 3, 2020 at 01:16 Ning Wang <[email protected]> wrote: >> >>> Looks like the exception is caused by pure virtual function calling. >>> >>> Both exceptions are from BaseClient. It seems like the Client object >>> doesn't have the virtual functions implemented, which is not expected. >>> >>> Another possibility is that the client object has been deleted hence the >>> vtable is not valid any more. This could be something you can check given >>> the last log shows "Stmgr stmgr-12 closed connection". >>> >>> >>> On Mon, Mar 2, 2020 at 12:20 PM Huijun Wu <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> We observed the Stmgr coredump, see the below logs. Anybody has any >>>> idea on >>>> this? Thanks. >>>> >>>> Best, >>>> Huijun >>>> >>>> ------------ >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: pure virtual >>>> method >>>> called >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: terminate called >>>> without an active exception >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: *** Aborted at >>>> 1582847383 (unix time) try "date -d @1582847383" if you are using GNU >>>> date >>>> *** >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: PC: @ >>>> 0x7f5bc42da277 __GI_raise >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: *** SIGABRT >>>> (@0xbcb0000b338) received by PID 45880 (TID 0x7f5bc54c1780) from PID >>>> 45880; >>>> stack trace: *** >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7f5bc50a76d0 (unknown) >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7f5bc42da277 __GI_raise >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7f5bc42db968 __GI_abort >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7f5bc48e77d5 __gnu_cxx::__verbose_terminate_handler() >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7f5bc48e5746 (unknown) >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7f5bc48e5773 std::terminate() >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7f5bc48e62df __cxa_pure_virtual >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4dfe1d BaseClient::OnConnect() >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4de404 EventLoopImpl::handleWriteCallback() >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4f0a9f event_process_active_single_queue >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4f115f event_base_loop >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x40c81f main >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7f5bc42c6445 __libc_start_main >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x41257c (unknown) >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x0 (unknown) >>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: >>>> ------------ >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: pure virtual >>>> method >>>> called >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: terminate called >>>> without an active exception >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: *** Aborted at >>>> 1582837876 (unix time) try "date -d @1582837876" if you are using GNU >>>> date >>>> *** >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: PC: @ >>>> 0x7fe9d8935277 __GI_raise >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: *** SIGABRT >>>> (@0xbcb0005176a) received by PID 333674 (TID 0x7fe9d9b1c780) from PID >>>> 333674; stack trace: *** >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7fe9d97026d0 (unknown) >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7fe9d8935277 __GI_raise >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7fe9d8936968 __GI_abort >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7fe9d8f427d5 __gnu_cxx::__verbose_terminate_handler() >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7fe9d8f40746 (unknown) >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7fe9d8f40773 std::terminate() >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7fe9d8f412df __cxa_pure_virtual >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4df98a BaseClient::Start_Base() >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x443dfa heron::stmgr::StMgrClient::OnReConnectTimer() >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4df6ed >>>> >>>> _ZNSt17_Function_handlerIFvN9EventLoop6StatusEEZN10BaseClient13AddTimer_BaseESt8functionIFvvEElEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4ded81 EventLoopImpl::handleTimerCallback() >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4dc840 EventLoopImpl::eventLoopImplTimerCallback() >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4f0a9f event_process_active_single_queue >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x4f115f event_base_loop >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x40c81f main >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x7fe9d8921445 __libc_start_main >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x41257c (unknown) >>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ >>>> 0x0 (unknown) >>>> [2020-02-27 21:11:17 +0000] [INFO]: stmgr-233 stdout: >>>> -------------------- >>>> I0227 21:11:16.813194 333674 stmgr-client.cpp:139] Will try to reconnect >>>> again after 1 seconds >>>> E0227 21:11:16.813606 333674 baseconnection.cpp:142] BufferEvent >>>> reported >>>> error on connection 0x2d2b720 >>>> I0227 21:11:16.813617 333674 stmgr-client.cpp:133] Stmgr stmgr-154 >>>> running >>>> at xxxxxxx:31831 closed connection with code Write error >>>> I0227 21:11:16.813670 333674 stmgr-client.cpp:139] Will try to reconnect >>>> again after 1 seconds >>>> E0227 21:11:16.814357 333674 baseconnection.cpp:142] BufferEvent >>>> reported >>>> error on connection 0x2d3aa00 >>>> I0227 21:11:16.814375 333674 stmgr-server.cpp:111] StMgrServer Got >>>> connection close of 0x2d3aa00 from yyyyyyyy:35270 >>>> I0227 21:11:16.814378 333674 stmgr-server.cpp:121] Stmgr stmgr-86 closed >>>> connection >>>> E0227 21:11:16.814669 333674 baseconnection.cpp:142] BufferEvent >>>> reported >>>> error on connection 0x2d3a220 >>>> I0227 21:11:16.814683 333674 stmgr-server.cpp:111] StMgrServer Got >>>> connection close of 0x2d3a220 from yyyyyyyy:33974 >>>> I0227 21:11:16.814685 333674 stmgr-server.cpp:121] Stmgr stmgr-154 >>>> closed >>>> connection >>>> E0227 21:11:16.816232 333674 baseconnection.cpp:142] BufferEvent >>>> reported >>>> error on connection 0x56ab800 >>>> I0227 21:11:16.816256 333674 stmgr-client.cpp:133] Stmgr stmgr-221 >>>> running >>>> at xxxxxxx:31294 closed connection with code Write error >>>> I0227 21:11:16.816263 333674 stmgr-client.cpp:139] Will try to reconnect >>>> again after 1 seconds >>>> E0227 21:11:16.817833 333674 baseconnection.cpp:142] BufferEvent >>>> reported >>>> error on connection 0x2d295e0 >>>> I0227 21:11:16.817849 333674 stmgr-client.cpp:133] Stmgr stmgr-12 >>>> running >>>> at xxxxxxx:31846 closed connection with code Write error >>>> I0227 21:11:16.817853 333674 stmgr-client.cpp:139] Will try to reconnect >>>> again after 1 seconds >>>> E0227 21:11:16.820160 333674 baseconnection.cpp:142] BufferEvent >>>> reported >>>> error on connection 0x2d39500 >>>> I0227 21:11:16.820192 333674 stmgr-server.cpp:111] StMgrServer Got >>>> connection close of 0x2d39500 from yyyyyyyy:41220 >>>> I0227 21:11:16.820196 333674 stmgr-server.cpp:121] Stmgr stmgr-12 closed >>>> connection >>>> >>> -- >> Thanks >> Xiaoyao >> > -- Thanks Xiaoyao
