[
https://issues.apache.org/jira/browse/MESOS-8346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298109#comment-16298109
]
Gilbert Song commented on MESOS-8346:
-------------------------------------
[~nfnt], seems like this is targeted for 1.5.0. Do we have an estimate when it
will land?
> Resubscription of a resource provider will crash the agent if its HTTP
> connection isn't closed
> ----------------------------------------------------------------------------------------------
>
> Key: MESOS-8346
> URL: https://issues.apache.org/jira/browse/MESOS-8346
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 1.5.0
> Reporter: Jan Schlicht
> Assignee: Jan Schlicht
> Priority: Blocker
> Labels: mesosphere
>
> A resource provider might resubscribe while its old HTTP connection wasn't
> properly closed. In that case an agent will crashm with, e.g., the following
> log:
> {noformat}
> I1219 13:33:51.937295 128610304 manager.cpp:570] Subscribing resource
> provider
> {"id":{"value":"8e71beef-796e-4bde-9257-952ed0f230a5"},"name":"test","type":"org.apache.mesos.rp.test"}
> I1219 13:33:51.937443 128610304 manager.cpp:134] Terminating resource
> provider 8e71beef-796e-4bde-9257-952ed0f230a5
> I1219 13:33:51.937760 128610304 manager.cpp:134] Terminating resource
> provider 8e71beef-796e-4bde-9257-952ed0f230a5
> E1219 13:33:51.937851 129683456 http_connection.hpp:445] End-Of-File received
> I1219 13:33:51.937865 131293184 slave.cpp:7105] Handling resource provider
> message 'DISCONNECT: resource provider 8e71beef-796e-4bde-9257-952ed0f230a5'
> I1219 13:33:51.937968 131293184 slave.cpp:7347] Forwarding new total
> resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]
> F1219 13:33:51.938052 132366336 manager.cpp:606] Check failed:
> resourceProviders.subscribed.contains(resourceProviderId)
> *** Check failure stack trace: ***
> E1219 13:33:51.938583 130756608 http_connection.hpp:445] End-Of-File received
> I1219 13:33:51.938987 129683456 hierarchical.cpp:669] Agent
> 0019c3fa-28c5-43a9-88d0-709eee271c62-S0 (172.18.8.13) updated with total
> resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]
> @ 0x1125380ef google::LogMessageFatal::~LogMessageFatal()
> @ 0x112534ae9 google::LogMessageFatal::~LogMessageFatal()
> I1219 13:33:51.939131 129683456 hierarchical.cpp:1517] Performed allocation
> for 1 agents in 61830ns
> I1219 13:33:51.945793 2646795072 slave.cpp:927] Agent terminating
> I1219 13:33:51.945955 129146880 master.cpp:1305] Agent
> 0019c3fa-28c5-43a9-88d0-709eee271c62-S0 at slave(1)@172.18.8.13:64430
> (172.18.8.13) disconnected
> I1219 13:33:51.945979 129146880 master.cpp:3364] Disconnecting agent
> 0019c3fa-28c5-43a9-88d0-709eee271c62-S0 at slave(1)@172.18.8.13:64430
> (172.18.8.13)
> I1219 13:33:51.946022 129146880 master.cpp:3383] Deactivating agent
> 0019c3fa-28c5-43a9-88d0-709eee271c62-S0 at slave(1)@172.18.8.13:64430
> (172.18.8.13)
> I1219 13:33:51.946081 131293184 hierarchical.cpp:766] Agent
> 0019c3fa-28c5-43a9-88d0-709eee271c62-S0 deactivated
> @ 0x115f2761d
> mesos::internal::ResourceProviderManagerProcess::subscribe()::$_2::operator()()
> @ 0x115f2977d
> _ZN5cpp176invokeIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS2_14HttpConnectionERKNS1_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEEDTclclsr3stdE7forwardIT_Efp_Espclsr3stdE7forwardIT0_Efp0_EEEOSG_DpOSH_
> @ 0x115f29740
> _ZN6lambda8internal7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS3_14HttpConnectionERKNS2_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEE13invoke_expandISC_NSt3__15tupleIJSG_EEENSK_IJEEEJLm0EEEEDTclsr5cpp17E6invokeclsr3stdE7forwardIT_Efp_Espcl6expandclsr3stdE3getIXT2_EEclsr3stdE7forwardIT0_Efp0_EEclsr3stdE7forwardIT1_Efp2_EEEEOSN_OSO_N5cpp1416integer_sequenceImJXspT2_EEEEOSP_
> @ 0x115f296bb
> _ZNO6lambda8internal7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS3_14HttpConnectionERKNS2_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEclIJEEEDTcl13invoke_expandclL_ZNSt3__14moveIRSC_EEONSJ_16remove_referenceIT_E4typeEOSN_EdtdefpT1fEclL_ZNSK_IRNSJ_5tupleIJSG_EEEEESQ_SR_EdtdefpT10bound_argsEcvN5cpp1416integer_sequenceImJLm0EEEE_Eclsr3stdE16forward_as_tuplespclsr3stdE7forwardIT_Efp_EEEEDpOSY_
> @ 0x115f2965d
> _ZN5cpp176invokeIN6lambda8internal7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS5_14HttpConnectionERKNS4_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEEJEEEDTclclsr3stdE7forwardIT_Efp_Espclsr3stdE7forwardIT0_Efp0_EEEOSK_DpOSL_
> @ 0x115f29631
> _ZN6lambda8internal6InvokeIvEclINS0_7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS6_14HttpConnectionERKNS5_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEEJEEEvOT_DpOT0_
> @ 0x115f29526
> _ZNO6lambda12CallableOnceIFvvEE10CallableFnINS_8internal7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS7_14HttpConnectionERKNS6_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEEEclEv
> @ 0x10b6ca690 _ZNO6lambda12CallableOnceIFvvEEclEv
> @ 0x10be09295
> _ZZN7process8internal8DispatchIvEclIN6lambda12CallableOnceIFvvEEEEEvRKNS_4UPIDEOT_ENKUlOS7_PNS_11ProcessBaseEE_clESD_SF_
> @ 0x10be09180
> _ZN5cpp176invokeIZN7process8internal8DispatchIvEclIN6lambda12CallableOnceIFvvEEEEEvRKNS1_4UPIDEOT_EUlOS9_PNS1_11ProcessBaseEE_JS9_SH_EEEDTclclsr3stdE7forwardISD_Efp_Espclsr3stdE7forwardIT0_Efp0_EEESE_DpOSJ_
> @ 0x10be0912b
> _ZN6lambda8internal7PartialIZN7process8internal8DispatchIvEclINS_12CallableOnceIFvvEEEEEvRKNS2_4UPIDEOT_EUlOS9_PNS2_11ProcessBaseEE_JS9_NSt3__112placeholders4__phILi1EEEEE13invoke_expandISI_NSJ_5tupleIJS9_SM_EEENSP_IJOSH_EEEJLm0ELm1EEEEDTclsr5cpp17E6invokeclsr3stdE7forwardISD_Efp_Espcl6expandclsr3stdE3getIXT2_EEclsr3stdE7forwardIT0_Efp0_EEclsr3stdE7forwardIT1_Efp2_EEEESE_OST_N5cpp1416integer_sequenceImJXspT2_EEEEOSU_
> @ 0x10be0905f
> _ZNO6lambda8internal7PartialIZN7process8internal8DispatchIvEclINS_12CallableOnceIFvvEEEEEvRKNS2_4UPIDEOT_EUlOS9_PNS2_11ProcessBaseEE_JS9_NSt3__112placeholders4__phILi1EEEEEclIJSH_EEEDTcl13invoke_expandclL_ZNSJ_4moveIRSI_EEONSJ_16remove_referenceISD_E4typeESE_EdtdefpT1fEclL_ZNSP_IRNSJ_5tupleIJS9_SM_EEEEESU_SE_EdtdefpT10bound_argsEcvN5cpp1416integer_sequenceImJLm0ELm1EEEE_Eclsr3stdE16forward_as_tuplespclsr3stdE7forwardIT_Efp_EEEEDpOS11_
> @ 0x10be08f4d
> _ZN5cpp176invokeIN6lambda8internal7PartialIZN7process8internal8DispatchIvEclINS1_12CallableOnceIFvvEEEEEvRKNS4_4UPIDEOT_EUlOSB_PNS4_11ProcessBaseEE_JSB_NSt3__112placeholders4__phILi1EEEEEEJSJ_EEEDTclclsr3stdE7forwardISF_Efp_Espclsr3stdE7forwardIT0_Efp0_EEESG_DpOSQ_
> @ 0x10be08f11
> _ZN6lambda8internal6InvokeIvEclINS0_7PartialIZN7process8internal8DispatchIvEclINS_12CallableOnceIFvvEEEEEvRKNS5_4UPIDEOT_EUlOSC_PNS5_11ProcessBaseEE_JSC_NSt3__112placeholders4__phILi1EEEEEEJSK_EEEvSH_DpOT0_
> @ 0x10be08d36
> _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8internal8DispatchIvEclINS0_IFvvEEEEEvRKNS1_4UPIDEOT_EUlOSE_S3_E_JSE_NSt3__112placeholders4__phILi1EEEEEEEclEOS3_
> @ 0x11fd64bc9
> _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEEclES3_
> @ 0x11fd64a69 process::ProcessBase::consume()
> @ 0x11fe20ac4
> _ZNO7process13DispatchEvent7consumeEPNS_13EventConsumerE
> @ 0x113c77819 process::ProcessBase::serve()
> @ 0x11fd5b8c9 process::ProcessManager::resume()
> @ 0x11fe8260b
> process::ProcessManager::init_threads()::$_1::operator()()
> @ 0x11fe82190
> _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN7process14ProcessManager12init_threadsEvE3$_1EEEEEPvSB_
> @ 0x7fff64da56c1 _pthread_body
> @ 0x7fff64da556d _pthread_start
> @ 0x7fff64da4c5d thread_start
> Abort trap: 6
> {noformat}
> This is due to a race condition in {{resource_provider/manager.cpp}} when
> handling closed HTTP connections of resource providers. If a resource
> provider resubscribes and its old HTTP connection is still open, the resource
> provider manager will close it. This is unexpected and will trigger closing
> the new HTTP connection which results in a failed {{CHECK}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
