Gilbert Song created MESOS-5587:
-----------------------------------
Summary: FullFrameworkWriter makes master segmentation fault.
Key: MESOS-5587
URL: https://issues.apache.org/jira/browse/MESOS-5587
Project: Mesos
Issue Type: Bug
Reporter: Gilbert Song
Assignee: Joerg Schad
Priority: Blocker
Fix For: 1.0.0
FullFrameworkWriter::operator() may take down the master. Here is the log:
{noformat}
Jun 09 02:28:42 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:42.147253 18633
master.cpp:5772] Sending 1 offers to framework
6d4248cd-2832-4152-b5d0-defbf36f6759-0001 (chronos) at
[email protected]:39285
Jun 09 02:28:42 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:42.148890 18637
master.cpp:4066] Processing DECLINE call for offers: [
7567c338-3ae5-4a84-bf5b-6a75a8a49341-O992 ] for framework
6d4248cd-2832-4152-b5d0-defbf36f6759-0001 (chronos) at
[email protected]:39285
Jun 09 02:28:42 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:42.639813 18632
http.cpp:483] HTTP GET for /master/state-summary from 10.10.0.180:45790 with
User-Agent='python-requests/2.6.0 CPython/3.4.2
Linux/3.10.0-327.10.1.el7.x86_64'
Jun 09 02:28:42 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:42.890702 18632
http.cpp:483] HTTP GET for /master/state from 10.10.0.181:33830 with
User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36'
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.139240 18639
http.cpp:483] HTTP GET for /master/state-summary from 10.10.0.181:33831 with
User-Agent='python-requests/2.6.0 CPython/3.4.2
Linux/3.10.0-327.18.2.el7.x86_64'
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.148582 18633
master.cpp:5772] Sending 1 offers to framework
4c6031e7-4cfd-4219-89b2-d19c7101e045-0001 (Long Lived Framework (C++))
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.150388 18635
http.cpp:483] HTTP POST for /master/api/v1/scheduler from 10.10.0.178:51645
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.150645 18635
master.cpp:3457] Processing ACCEPT call for offers: [
7567c338-3ae5-4a84-bf5b-6a75a8a49341-O993 ] on agent
091e9c3f-8a01-4890-8790-48b75fd81b40-S0 at slave(1)@10.10.0.20:5051
(10.10.0.20) for framework 4c6031e7-4cfd-4219-89b2-d19c7101e045-0001 (Long
Lived Framework (C++))
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.151268 18635
master.hpp:178] Adding task 5699 with resources cpus(*):0.001; mem(*):1 on
agent 091e9c3f-8a01-4890-8790-48b75fd81b40-S0 (10.10.0.20)
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.151322 18635
master.cpp:3946] Launching task 5699 of framework
4c6031e7-4cfd-4219-89b2-d19c7101e045-0001 (Long Lived Framework (C++)) with
resources cpus(*):0.001; mem(*):1 on agent
091e9c3f-8a01-4890-8790-48b75fd81b40-S0 at slave(1)@10.10.0.20:5051 (10.10.0.20)
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.160475 18635
master.cpp:5211] Status update TASK_RUNNING (UUID:
3f651ba8-7c80-4ac0-ae18-579371ec82d5) for task 5699 of framework
4c6031e7-4cfd-4219-89b2-d19c7101e045-0001 from agent
091e9c3f-8a01-4890-8790-48b75fd81b40-S0 at slave(1)@10.10.0.20:5051 (10.10.0.20)
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.160516 18635
master.cpp:5259] Forwarding status update TASK_RUNNING (UUID:
3f651ba8-7c80-4ac0-ae18-579371ec82d5) for task 5699 of framework
4c6031e7-4cfd-4219-89b2-d19c7101e045-0001
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.160645 18635
master.cpp:6871] Updating the state of task 5699 of framework
4c6031e7-4cfd-4219-89b2-d19c7101e045-0001 (latest state: TASK_RUNNING, status
update state: TASK_RUNNING)
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.161842 18639
http.cpp:483] HTTP POST for /master/api/v1/scheduler from 10.10.0.178:51645
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.161912 18639
master.cpp:4365] Processing ACKNOWLEDGE call
3f651ba8-7c80-4ac0-ae18-579371ec82d5 for task 5699 of framework
4c6031e7-4cfd-4219-89b2-d19c7101e045-0001 (Long Lived Framework (C++)) on agent
091e9c3f-8a01-4890-8790-48b75fd81b40-S0
Jun 09 02:28:43 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:43.556354 18639
http.cpp:483] HTTP GET for /master/state-summary from 10.10.0.93:55478 with
User-Agent='python-requests/2.6.0 CPython/3.4.2
Linux/3.10.0-327.10.1.el7.x86_64'
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:44.649113 18635
http.cpp:483] HTTP GET for /master/state-summary from 10.10.0.180:45793 with
User-Agent='python-requests/2.6.0 CPython/3.4.2
Linux/3.10.0-327.10.1.el7.x86_64'
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:44.660256 18636
http.cpp:483] HTTP GET for /master/state.json from 10.10.0.180:45794
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:44.676964 18632
http.cpp:483] HTTP GET for /master/state.json from 10.10.0.180:45795
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: I0609 02:28:44.677815 18638
http.cpp:483] HTTP GET for /master/state.json from 10.10.0.180:45796
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: *** Aborted at 1465439324
(unix time) try "date -d @1465439324" if you are using GNU date ***
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: PC: @ 0x7f35ef5b6e33
_ZZNK5mesos8internal6master19FullFrameworkWriterclEPN4JSON12ObjectWriterEENKUlPNS3_11ArrayWriterEE0_clES7_
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: *** SIGSEGV (@0x8) received
by PID 18627 (TID 0x7f35eb146700) from PID 8; stack trace: ***
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35eda79100
(unknown)
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5b6e33
_ZZNK5mesos8internal6master19FullFrameworkWriterclEPN4JSON12ObjectWriterEENKUlPNS3_11ArrayWriterEE0_clES7_
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5b7035
_ZNSt17_Function_handlerIFvPSoEZN4JSON8internal7jsonifyIZNK5mesos8internal6master19FullFrameworkWriterclEPNS2_12ObjectWriterEEUlPNS2_11ArrayWriterEE0_vEESt8functionIS1_ERKT_NS3_6PreferEEUlS0_E_E9_M_invokeERKSt9_Any_dataS0_
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5d0137
mesos::internal::master::FullFrameworkWriter::operator()()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5d0945
_ZNSt17_Function_handlerIFvPSoEZN4JSON8internal7jsonifyIN5mesos8internal6master19FullFrameworkWriterEvEESt8functionIS1_ERKT_NS3_6PreferEEUlS0_E_E9_M_invokeERKSt9_Any_dataS0_
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef58a081
_ZNSt17_Function_handlerIFvPSoEZN4JSON8internal7jsonifyIZZZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionISsEENKUlRKSt5tupleIINSA_5OwnedINS5_14ObjectApproverEEESM_SM_EEE_clESP_ENKUlPNS2_12ObjectWriterEE_clESS_EUlPNS2_11ArrayWriterEE1_vEESt8functionIS1_ERKT_NS3_6PreferEEUlS0_E_E9_M_invokeERKSt9_Any_dataS0_
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5a6a08
_ZZZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionISsEENKUlRKSt5tupleIINS4_5OwnedINS_14ObjectApproverEEESG_SG_EEE_clESJ_ENKUlPN4JSON12ObjectWriterEE_clESN_
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5a7705
_ZNSt17_Function_handlerIFvPSoEZN4JSON8internal7jsonifyIZZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionISsEENKUlRKSt5tupleIINSA_5OwnedINS5_14ObjectApproverEEESM_SM_EEE_clESP_EUlPNS2_12ObjectWriterEE_vEESt8functionIS1_ERKT_NS3_6PreferEEUlS0_E_E9_M_invokeERKSt9_Any_dataS0_
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35efe45a10
process::http::OK::OK()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef58fb24
_ZNSt17_Function_handlerIFN7process6FutureINS0_4http8ResponseEEERKSt5tupleIINS0_5OwnedIN5mesos14ObjectApproverEEES9_S9_EEEZNKS7_8internal6master6Master4Http5stateERKNS2_7RequestERK6OptionISsEEUlSC_E_E9_M_invokeERKSt9_Any_dataSC_
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5c539e
_ZN7process8internal5thenfISt5tupleIJNS_5OwnedIN5mesos14ObjectApproverEEES6_S6_EENS_4http8ResponseEEEvRKSt8functionIFNS_6FutureIT0_EERKT_EERKSt10shared_ptrINS_7PromiseISC_EEERKNSB_ISE_EE
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5c3e06
process::internal::run<>()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5c8aca
process::Future<>::_set<>()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5c9e4c
_ZN7process7PromiseISt5tupleIJNS_5OwnedIN5mesos14ObjectApproverEEES5_S5_EEE9associateERKNS_6FutureIS6_EE
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5ca27a
process::internal::thenf<>()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5c15b6
process::internal::run<>()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5c379b
process::Future<>::_set<>()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ef5cda4e
process::internal::CollectProcess<>::waited()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35efe951c1
process::ProcessManager::resume()
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35efe954c7
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ee253220
(unknown)
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35eda71dc5
start_thread
Jun 09 02:28:44 ip-10-10-0-180 mesos-master[18627]: @ 0x7f35ed79eced __clone
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
