Observed that, restarting single container will lead to all stmgr failure and restarting. Thus killing a single stmgr should reproduce this issue.
On Tue, Mar 3, 2020 at 10:58 AM Ning Wang <[email protected]> wrote: > Ok. The log shows that the connection is being closed. So it just looks > like a network issue causing the connection to be unstable. > > Looks like a racing condition to me if this theory is correct. You may need > to review the object and add a mutex around the critical sections. > > On Tue, Mar 3, 2020 at 10:07 AM Xiaoyao Qian <[email protected]> wrote: > > > During normal running state. And it happens only to one data center.. > > > > On Tue, Mar 3, 2020 at 9:33 AM Ning Wang <[email protected]> wrote: > > > >> Yeah. Segfault is more likely in this case. But maybe not for the object > >> itself but for the vtable pointer. > >> > >> Pure function call might be possible though I think. It all depends on > >> the 4 bytes in the address. It could happen If a new object of a > different > >> type has been created in the place or an object is moved to the memory > >> block. > >> > >> More investigation is necessary. > >> > >> When does it happen? normal running state or when topo or instance is > >> shutting down? > >> > >> > >> On Tue, Mar 3, 2020 at 8:34 AM Xiaoyao Qian <[email protected]> wrote: > >> > >>> I agreed with the second possibility, but wouldn’t it cause segfault if > >>> the object has been deleted but the method is still invoked? > >>> > >>> On Tue, Mar 3, 2020 at 01:16 Ning Wang <[email protected]> wrote: > >>> > >>>> Looks like the exception is caused by pure virtual function calling. > >>>> > >>>> Both exceptions are from BaseClient. It seems like the Client object > >>>> doesn't have the virtual functions implemented, which is not expected. > >>>> > >>>> Another possibility is that the client object has been deleted hence > >>>> the vtable is not valid any more. This could be something you can > check > >>>> given the last log shows "Stmgr stmgr-12 closed connection". > >>>> > >>>> > >>>> On Mon, Mar 2, 2020 at 12:20 PM Huijun Wu <[email protected]> > >>>> wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> We observed the Stmgr coredump, see the below logs. Anybody has any > >>>>> idea on > >>>>> this? Thanks. > >>>>> > >>>>> Best, > >>>>> Huijun > >>>>> > >>>>> ------------ > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: pure virtual > >>>>> method > >>>>> called > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: terminate > called > >>>>> without an active exception > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: *** Aborted at > >>>>> 1582847383 (unix time) try "date -d @1582847383" if you are using GNU > >>>>> date > >>>>> *** > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: PC: @ > >>>>> 0x7f5bc42da277 __GI_raise > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: *** SIGABRT > >>>>> (@0xbcb0000b338) received by PID 45880 (TID 0x7f5bc54c1780) from PID > >>>>> 45880; > >>>>> stack trace: *** > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7f5bc50a76d0 (unknown) > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7f5bc42da277 __GI_raise > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7f5bc42db968 __GI_abort > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7f5bc48e77d5 __gnu_cxx::__verbose_terminate_handler() > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7f5bc48e5746 (unknown) > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7f5bc48e5773 std::terminate() > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7f5bc48e62df __cxa_pure_virtual > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4dfe1d BaseClient::OnConnect() > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4de404 EventLoopImpl::handleWriteCallback() > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4f0a9f event_process_active_single_queue > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4f115f event_base_loop > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x40c81f main > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7f5bc42c6445 __libc_start_main > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x41257c (unknown) > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x0 (unknown) > >>>>> [2020-02-27 23:49:43 +0000] [INFO]: stmgr-233 stdout: > >>>>> ------------ > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: pure virtual > >>>>> method > >>>>> called > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: terminate > called > >>>>> without an active exception > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: *** Aborted at > >>>>> 1582837876 (unix time) try "date -d @1582837876" if you are using GNU > >>>>> date > >>>>> *** > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: PC: @ > >>>>> 0x7fe9d8935277 __GI_raise > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: *** SIGABRT > >>>>> (@0xbcb0005176a) received by PID 333674 (TID 0x7fe9d9b1c780) from PID > >>>>> 333674; stack trace: *** > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7fe9d97026d0 (unknown) > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7fe9d8935277 __GI_raise > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7fe9d8936968 __GI_abort > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7fe9d8f427d5 __gnu_cxx::__verbose_terminate_handler() > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7fe9d8f40746 (unknown) > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7fe9d8f40773 std::terminate() > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7fe9d8f412df __cxa_pure_virtual > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4df98a BaseClient::Start_Base() > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x443dfa heron::stmgr::StMgrClient::OnReConnectTimer() > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4df6ed > >>>>> > >>>>> > _ZNSt17_Function_handlerIFvN9EventLoop6StatusEEZN10BaseClient13AddTimer_BaseESt8functionIFvvEElEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_ > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4ded81 EventLoopImpl::handleTimerCallback() > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4dc840 EventLoopImpl::eventLoopImplTimerCallback() > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4f0a9f event_process_active_single_queue > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x4f115f event_base_loop > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x40c81f main > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x7fe9d8921445 __libc_start_main > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x41257c (unknown) > >>>>> [2020-02-27 21:11:16 +0000] [INFO]: stmgr-233 stdout: @ > >>>>> 0x0 (unknown) > >>>>> [2020-02-27 21:11:17 +0000] [INFO]: stmgr-233 stdout: > >>>>> -------------------- > >>>>> I0227 21:11:16.813194 333674 stmgr-client.cpp:139] Will try to > >>>>> reconnect > >>>>> again after 1 seconds > >>>>> E0227 21:11:16.813606 333674 baseconnection.cpp:142] BufferEvent > >>>>> reported > >>>>> error on connection 0x2d2b720 > >>>>> I0227 21:11:16.813617 333674 stmgr-client.cpp:133] Stmgr stmgr-154 > >>>>> running > >>>>> at xxxxxxx:31831 closed connection with code Write error > >>>>> I0227 21:11:16.813670 333674 stmgr-client.cpp:139] Will try to > >>>>> reconnect > >>>>> again after 1 seconds > >>>>> E0227 21:11:16.814357 333674 baseconnection.cpp:142] BufferEvent > >>>>> reported > >>>>> error on connection 0x2d3aa00 > >>>>> I0227 21:11:16.814375 333674 stmgr-server.cpp:111] StMgrServer Got > >>>>> connection close of 0x2d3aa00 from yyyyyyyy:35270 > >>>>> I0227 21:11:16.814378 333674 stmgr-server.cpp:121] Stmgr stmgr-86 > >>>>> closed > >>>>> connection > >>>>> E0227 21:11:16.814669 333674 baseconnection.cpp:142] BufferEvent > >>>>> reported > >>>>> error on connection 0x2d3a220 > >>>>> I0227 21:11:16.814683 333674 stmgr-server.cpp:111] StMgrServer Got > >>>>> connection close of 0x2d3a220 from yyyyyyyy:33974 > >>>>> I0227 21:11:16.814685 333674 stmgr-server.cpp:121] Stmgr stmgr-154 > >>>>> closed > >>>>> connection > >>>>> E0227 21:11:16.816232 333674 baseconnection.cpp:142] BufferEvent > >>>>> reported > >>>>> error on connection 0x56ab800 > >>>>> I0227 21:11:16.816256 333674 stmgr-client.cpp:133] Stmgr stmgr-221 > >>>>> running > >>>>> at xxxxxxx:31294 closed connection with code Write error > >>>>> I0227 21:11:16.816263 333674 stmgr-client.cpp:139] Will try to > >>>>> reconnect > >>>>> again after 1 seconds > >>>>> E0227 21:11:16.817833 333674 baseconnection.cpp:142] BufferEvent > >>>>> reported > >>>>> error on connection 0x2d295e0 > >>>>> I0227 21:11:16.817849 333674 stmgr-client.cpp:133] Stmgr stmgr-12 > >>>>> running > >>>>> at xxxxxxx:31846 closed connection with code Write error > >>>>> I0227 21:11:16.817853 333674 stmgr-client.cpp:139] Will try to > >>>>> reconnect > >>>>> again after 1 seconds > >>>>> E0227 21:11:16.820160 333674 baseconnection.cpp:142] BufferEvent > >>>>> reported > >>>>> error on connection 0x2d39500 > >>>>> I0227 21:11:16.820192 333674 stmgr-server.cpp:111] StMgrServer Got > >>>>> connection close of 0x2d39500 from yyyyyyyy:41220 > >>>>> I0227 21:11:16.820196 333674 stmgr-server.cpp:121] Stmgr stmgr-12 > >>>>> closed > >>>>> connection > >>>>> > >>>> -- > >>> Thanks > >>> Xiaoyao > >>> > >> > > > > -- > > Thanks > > Xiaoyao > > >
