Anyone know what std::system_error with 'Invalid argument' might mean? Or why we're getting a stack trace entirely of '(Unknown)'?
I've actually seen the same crash in a scale test cluster, would be great if anyone has any ideas here. On Wed, Aug 16, 2017 at 6:00 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See <https://builds.apache.org/job/Mesos-Buildbot/BUILDTOOL=auto > tools,COMPILER=clang,CONFIGURATION=--verbose%20--enable- > libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_ > VERBOSE=1,OS=ubuntu%3A14.04,label_exp=(ubuntu)&&(!ubuntu- > us1)&&(!ubuntu-eu2)/4118/display/redirect?page=changes> > > Changes: > > [anand] Fixed a bug around `kill()` not removing active containers. > > ------------------------------------------ > [...truncated 21.31 MB...] > I0817 01:00:05.801321 6737 hierarchical.cpp:1943] No allocations performed > I0817 01:00:05.801424 6737 hierarchical.cpp:2033] No inverse offers to > send out! > I0817 01:00:05.801476 6737 hierarchical.cpp:1486] Performed allocation > for 1 agents in 303266ns > I0817 01:00:06.316730 6724 slave.cpp:4273] Cleaning up un-reregistered > executors > I0817 01:00:06.316900 6724 slave.cpp:6210] Finished recovery > I0817 01:00:06.317662 6724 slave.cpp:6392] Querying resource estimator > for oversubscribable resources > I0817 01:00:06.317929 6745 status_update_manager.cpp:177] Pausing sending > status updates > I0817 01:00:06.317962 6725 slave.cpp:971] New master detected at > master@172.17.0.8:40444 > I0817 01:00:06.318066 6725 slave.cpp:1006] Detecting new master > I0817 01:00:06.318202 6725 slave.cpp:6406] Received oversubscribable > resources {} from the resource estimator > I0817 01:00:06.324371 6723 slave.cpp:1033] Authenticating with master > master@172.17.0.8:40444 > I0817 01:00:06.324511 6723 slave.cpp:1044] Using default CRAM-MD5 > authenticatee > I0817 01:00:06.324862 6740 authenticatee.cpp:121] Creating new client > SASL connection > I0817 01:00:06.325322 6729 master.cpp:7837] Authenticating slave(478)@ > 172.17.0.8:40444 > I0817 01:00:06.325479 6730 authenticator.cpp:414] Starting authentication > session for crammd5-authenticatee(993)@172.17.0.8:40444 > I0817 01:00:06.325743 6722 authenticator.cpp:98] Creating new server SASL > connection > I0817 01:00:06.326010 6739 authenticatee.cpp:213] Received SASL > authentication mechanisms: CRAM-MD5 > I0817 01:00:06.326045 6739 authenticatee.cpp:239] Attempting to > authenticate with mechanism 'CRAM-MD5' > I0817 01:00:06.326155 6739 authenticator.cpp:204] Received SASL > authentication start > I0817 01:00:06.326222 6739 authenticator.cpp:326] Authentication requires > more steps > I0817 01:00:06.326323 6728 authenticatee.cpp:259] Received SASL > authentication step > I0817 01:00:06.326432 6727 authenticator.cpp:232] Received SASL > authentication step > I0817 01:00:06.326464 6727 auxprop.cpp:109] Request to lookup properties > for user: 'test-principal' realm: '45197c1facb6' server FQDN: > '45197c1facb6' SASL_AUXPROP_VERIFY_AGAINST_HASH: false > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false > I0817 01:00:06.326478 6727 auxprop.cpp:181] Looking up auxiliary property > '*userPassword' > I0817 01:00:06.326519 6727 auxprop.cpp:181] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I0817 01:00:06.326539 6727 auxprop.cpp:109] Request to lookup properties > for user: 'test-principal' realm: '45197c1facb6' server FQDN: > '45197c1facb6' SASL_AUXPROP_VERIFY_AGAINST_HASH: false > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true > I0817 01:00:06.326550 6727 auxprop.cpp:131] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I0817 01:00:06.326558 6727 auxprop.cpp:131] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I0817 01:00:06.326575 6727 authenticator.cpp:318] Authentication success > I0817 01:00:06.326709 6726 authenticatee.cpp:299] Authentication success > I0817 01:00:06.326738 6742 master.cpp:7867] Successfully authenticated > principal 'test-principal' at slave(478)@172.17.0.8:40444 > I0817 01:00:06.326762 6734 authenticator.cpp:432] Authentication session > cleanup for crammd5-authenticatee(993)@172.17.0.8:40444 > I0817 01:00:06.326993 6737 slave.cpp:1128] Successfully authenticated > with master master@172.17.0.8:40444 > I0817 01:00:06.327518 6737 slave.cpp:1572] Will retry registration in > 14.251184ms if necessary > I0817 01:00:06.328802 6736 master.cpp:6012] Received re-register agent > message from agent 56b17556-5965-44ca-be07-44edbcb8751c-S0 at slave(478)@ > 172.17.0.8:40444 (45197c1facb6) > I0817 01:00:06.328842 6736 master.cpp:3803] Authorizing agent with > principal 'test-principal' > I0817 01:00:06.329376 6736 master.cpp:6081] Authorized re-registration of > agent 56b17556-5965-44ca-be07-44edbcb8751c-S0 at slave(478)@ > 172.17.0.8:40444 (45197c1facb6) > I0817 01:00:06.329483 6736 master.cpp:6146] Re-registering agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 at slave(477)@172.17.0.8:40444 > (45197c1facb6) > I0817 01:00:06.329792 6730 slave.cpp:1275] Re-registered with master > master@172.17.0.8:40444 > I0817 01:00:06.329824 6722 hierarchical.cpp:678] Agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 reactivated > I0817 01:00:06.329856 6736 master.cpp:6586] Sending updated checkpointed > resources {} to agent 56b17556-5965-44ca-be07-44edbcb8751c-S0 at > slave(478)@172.17.0.8:40444 (45197c1facb6) > I0817 01:00:06.329866 6722 status_update_manager.cpp:184] Resuming > sending status updates > I0817 01:00:06.329885 6730 slave.cpp:1312] Forwarding total > oversubscribed resources {} > W0817 01:00:06.329907 6722 status_update_manager.cpp:191] Resending > status update TASK_RUNNING (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) > for task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.329957 6722 status_update_manager.cpp:377] Forwarding > update TASK_RUNNING (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 to the agent > I0817 01:00:06.330029 6736 master.cpp:6688] Received update of agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 at slave(478)@172.17.0.8:40444 > (45197c1facb6) with total oversubscribed resources {} > I0817 01:00:06.330075 6730 slave.cpp:3330] Updating info for framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 with pid updated to > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444 > I0817 01:00:06.330152 6730 slave.cpp:7084] Checkpointing FrameworkInfo to > '/tmp/SlaveRecoveryTest_0_Reboot_kqojFE/meta/slaves/56b17556 > -5965-44ca-be07-44edbcb8751c-S0/frameworks/56b17556-5965- > 44ca-be07-44edbcb8751c-0000/framework.info' > I0817 01:00:06.330344 6728 hierarchical.cpp:660] Agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 (45197c1facb6) updated with total > resources cpus:2; mem:1024; disk:1024; ports:[31000-32000] > I0817 01:00:06.330607 6730 slave.cpp:7095] Checkpointing framework pid ' > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444' to > '/tmp/SlaveRecoveryTest_0_Reboot_kqojFE/meta/slaves/56b17556 > -5965-44ca-be07-44edbcb8751c-S0/frameworks/56b17556-5965- > 44ca-be07-44edbcb8751c-0000/framework.pid' > I0817 01:00:06.330912 6727 status_update_manager.cpp:184] Resuming > sending status updates > W0817 01:00:06.330937 6727 status_update_manager.cpp:191] Resending > status update TASK_RUNNING (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) > for task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.330953 6730 slave.cpp:3394] Ignoring new checkpointed > resources identical to the current version: {} > I0817 01:00:06.330968 6727 status_update_manager.cpp:377] Forwarding > update TASK_RUNNING (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 to the agent > I0817 01:00:06.331070 6730 slave.cpp:4797] Forwarding the update > TASK_RUNNING (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 to master@172.17.0.8:40444 > I0817 01:00:06.331285 6730 slave.cpp:4797] Forwarding the update > TASK_RUNNING (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 to master@172.17.0.8:40444 > I0817 01:00:06.331362 6726 master.cpp:6846] Status update TASK_RUNNING > (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 from agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 at slave(478)@172.17.0.8:40444 > (45197c1facb6) > I0817 01:00:06.331408 6726 master.cpp:6908] Forwarding status update > TASK_RUNNING (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.331564 6726 master.cpp:8933] Updating the state of task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (latest state: TASK_FAILED, > status update state: TASK_RUNNING) > I0817 01:00:06.331709 6744 sched.cpp:1041] Scheduler::statusUpdate took > 41560ns > I0817 01:00:06.332159 6726 master.cpp:6846] Status update TASK_RUNNING > (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 from agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 at slave(478)@172.17.0.8:40444 > (45197c1facb6) > I0817 01:00:06.332201 6726 master.cpp:6908] Forwarding status update > TASK_RUNNING (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.332340 6726 master.cpp:8933] Updating the state of task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (latest state: TASK_FAILED, > status update state: TASK_RUNNING) > I0817 01:00:06.332479 6733 sched.cpp:1041] Scheduler::statusUpdate took > 23192ns > I0817 01:00:06.332495 6726 master.cpp:5477] Processing ACKNOWLEDGE call > 6298f06a-4d56-49b0-ae14-12a4fd774566 for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (default) at > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444 on agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 > I0817 01:00:06.332500 6742 hierarchical.cpp:1161] Recovered > cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; > ports(allocated: *):[31000-32000] (total: cpus:2; mem:1024; disk:1024; > ports:[31000-32000], allocated: {}) on agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 > from framework 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.332691 6726 master.cpp:5477] Processing ACKNOWLEDGE call > 6298f06a-4d56-49b0-ae14-12a4fd774566 for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (default) at > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444 on agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 > I0817 01:00:06.332728 6734 status_update_manager.cpp:395] Received status > update acknowledgement (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for > task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.332854 6734 status_update_manager.cpp:834] Checkpointing > ACK for status update TASK_RUNNING (UUID: > 6298f06a-4d56-49b0-ae14-12a4fd774566) > for task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.332988 6734 status_update_manager.cpp:377] Forwarding > update TASK_FAILED (UUID: b29e6cb2-8461-43c6-b252-a7b4619bd9d4) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 to the agent > I0817 01:00:06.333187 6724 slave.cpp:4797] Forwarding the update > TASK_FAILED (UUID: b29e6cb2-8461-43c6-b252-a7b4619bd9d4) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 to master@172.17.0.8:40444 > I0817 01:00:06.333245 6734 status_update_manager.cpp:395] Received status > update acknowledgement (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for > task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > W0817 01:00:06.333329 6734 status_update_manager.cpp:763] Duplicate > status update acknowledgment (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) > for update TASK_FAILED (UUID: b29e6cb2-8461-43c6-b252-a7b4619bd9d4) for > task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.333354 6724 slave.cpp:3609] Status update manager > successfully handled status update acknowledgement (UUID: > 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > E0817 01:00:06.333447 6724 slave.cpp:3602] Failed to handle status update > acknowledgement (UUID: 6298f06a-4d56-49b0-ae14-12a4fd774566) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000: Duplicate acknowledgement > I0817 01:00:06.333453 6738 master.cpp:6846] Status update TASK_FAILED > (UUID: b29e6cb2-8461-43c6-b252-a7b4619bd9d4) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 from agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 at slave(478)@172.17.0.8:40444 > (45197c1facb6) > I0817 01:00:06.333504 6738 master.cpp:6908] Forwarding status update > TASK_FAILED (UUID: b29e6cb2-8461-43c6-b252-a7b4619bd9d4) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.333643 6738 master.cpp:8933] Updating the state of task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (latest state: TASK_FAILED, > status update state: TASK_FAILED) > I0817 01:00:06.333762 6732 sched.cpp:1041] Scheduler::statusUpdate took > 39113ns > I0817 01:00:06.333951 6745 master.cpp:5477] Processing ACKNOWLEDGE call > b29e6cb2-8461-43c6-b252-a7b4619bd9d4 for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (default) at > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444 on agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 > I0817 01:00:06.334004 6745 master.cpp:9027] Removing task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 with resources > [{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"va > lue":2.0},"type":"SCALAR"},{"allocation_info":{"role":"*"}, > "name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{" > allocation_info":{"role":"*"},"name":"disk","scalar":{"value > ":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"}, > "name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] > of framework 56b17556-5965-44ca-be07-44edbcb8751c-0000 on agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 at slave(478)@172.17.0.8:40444 > (45197c1facb6) > I0817 01:00:06.334609 6735 status_update_manager.cpp:395] Received status > update acknowledgement (UUID: b29e6cb2-8461-43c6-b252-a7b4619bd9d4) for > task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.334683 6735 status_update_manager.cpp:834] Checkpointing > ACK for status update TASK_FAILED (UUID: b29e6cb2-8461-43c6-b252-a7b4619bd9d4) > for task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.334759 6735 status_update_manager.cpp:531] Cleaning up > status update stream for task 334ab1cb-44b2-4676-b2b5-0329c383ac33 of > framework 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.335131 6723 slave.cpp:3609] Status update manager > successfully handled status update acknowledgement (UUID: > b29e6cb2-8461-43c6-b252-a7b4619bd9d4) for task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.335178 6723 slave.cpp:7561] Completing task > 334ab1cb-44b2-4676-b2b5-0329c383ac33 > I0817 01:00:06.335224 6723 slave.cpp:5429] Cleaning up executor > '334ab1cb-44b2-4676-b2b5-0329c383ac33' of framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 at executor(1)@172.17.0.8:40576 > I0817 01:00:06.335512 6740 gc.cpp:59] Scheduling > '/tmp/SlaveRecoveryTest_0_Reboot_kqojFE/slaves/56b17556-5965 > -44ca-be07-44edbcb8751c-S0/frameworks/56b17556-5965-44ca- > be07-44edbcb8751c-0000/executors/334ab1cb-44b2-4676-b2b5- > 0329c383ac33/runs/85315bbe-bb58-416d-853d-8c17b4a61361' for gc > 6.99999611762667days in the future > I0817 01:00:06.335686 6729 gc.cpp:59] Scheduling > '/tmp/SlaveRecoveryTest_0_Reboot_kqojFE/slaves/56b17556-5965 > -44ca-be07-44edbcb8751c-S0/frameworks/56b17556-5965-44ca- > be07-44edbcb8751c-0000/executors/334ab1cb-44b2-4676-b2b5-0329c383ac33' > for gc 6.99999611555556days in the future > I0817 01:00:06.335827 6729 gc.cpp:59] Scheduling > '/tmp/SlaveRecoveryTest_0_Reboot_kqojFE/meta/slaves/56b17556 > -5965-44ca-be07-44edbcb8751c-S0/frameworks/56b17556-5965- > 44ca-be07-44edbcb8751c-0000/executors/334ab1cb-44b2-4676- > b2b5-0329c383ac33/runs/85315bbe-bb58-416d-853d-8c17b4a61361' for gc > 6.99999611430519days in the future > I0817 01:00:06.335861 6723 slave.cpp:5525] Cleaning up framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.335929 6729 gc.cpp:59] Scheduling > '/tmp/SlaveRecoveryTest_0_Reboot_kqojFE/meta/slaves/56b17556 > -5965-44ca-be07-44edbcb8751c-S0/frameworks/56b17556-5965- > 44ca-be07-44edbcb8751c-0000/executors/334ab1cb-44b2-4676-b2b5-0329c383ac33' > for gc 6.99999611335704days in the future > I0817 01:00:06.335980 6731 status_update_manager.cpp:285] Closing status > update streams for framework 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.336024 6729 gc.cpp:59] Scheduling > '/tmp/SlaveRecoveryTest_0_Reboot_kqojFE/slaves/56b17556-5965 > -44ca-be07-44edbcb8751c-S0/frameworks/56b17556-5965-44ca-be07-44edbcb8751c-0000' > for gc 6.99999611171259days in the future > I0817 01:00:06.336120 6729 gc.cpp:59] Scheduling > '/tmp/SlaveRecoveryTest_0_Reboot_kqojFE/meta/slaves/56b17556 > -5965-44ca-be07-44edbcb8751c-S0/frameworks/56b17556-5965-44ca-be07-44edbcb8751c-0000' > for gc 6.9999961109363days in the future > I0817 01:00:06.802976 6742 hierarchical.cpp:2033] No inverse offers to > send out! > I0817 01:00:06.803063 6742 hierarchical.cpp:1486] Performed allocation > for 1 agents in 1.010273ms > I0817 01:00:06.803483 6733 master.cpp:7667] Sending 1 offers to framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (default) at > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444 > I0817 01:00:06.804096 6741 sched.cpp:933] Scheduler::resourceOffers took > 135353ns > I0817 01:00:06.804899 6704 sched.cpp:2021] Asked to stop the driver > I0817 01:00:06.804999 6738 sched.cpp:1203] Stopping framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.805320 6745 master.cpp:8348] Processing TEARDOWN call for > framework 56b17556-5965-44ca-be07-44edbcb8751c-0000 (default) at > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444 > I0817 01:00:06.805352 6745 master.cpp:8360] Removing framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (default) at > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444 > I0817 01:00:06.805367 6745 master.cpp:3264] Deactivating framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 (default) at > scheduler-01d4e1d6-01e5-43cf-9fb4-ca558d87d299@172.17.0.8:40444 > I0817 01:00:06.805480 6740 hierarchical.cpp:412] Deactivated framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.805881 6745 master.cpp:9164] Removing offer > 56b17556-5965-44ca-be07-44edbcb8751c-O1 > I0817 01:00:06.806056 6739 slave.cpp:3159] Asked to shut down framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 by master@172.17.0.8:40444 > I0817 01:00:06.806116 6739 slave.cpp:3174] Cannot shut down unknown > framework 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.806344 6740 hierarchical.cpp:1161] Recovered > cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; > ports(allocated: *):[31000-32000] (total: cpus:2; mem:1024; disk:1024; > ports:[31000-32000], allocated: {}) on agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 > from framework 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.809447 6740 hierarchical.cpp:355] Removed framework > 56b17556-5965-44ca-be07-44edbcb8751c-0000 > I0817 01:00:06.809541 6730 slave.cpp:843] Agent terminating > I0817 01:00:06.814934 6704 master.cpp:1160] Master terminating > I0817 01:00:06.815558 6739 hierarchical.cpp:626] Removed agent > 56b17556-5965-44ca-be07-44edbcb8751c-S0 > [ OK ] SlaveRecoveryTest/0.Reboot (4030 ms) > [ RUN ] SlaveRecoveryTest/0.RebootWithSlaveInfoMismatch > I0817 01:00:06.824440 6704 cluster.cpp:162] Creating default 'local' > authorizer > I0817 01:00:06.827071 6734 master.cpp:442] Master > 66249101-3204-4bec-9632-4a28a9d4b6e5 (45197c1facb6) started on > 172.17.0.8:40444 > I0817 01:00:06.827113 6734 master.cpp:444] Flags at startup: --acls="" > --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" > --allocation_interval="1secs" --allocator="HierarchicalDRF" > --authenticate_agents="true" --authenticate_frameworks="true" > --authenticate_http_frameworks="true" --authenticate_http_readonly="true" > --authenticate_http_readwrite="true" --authenticators="crammd5" > --authorizers="local" --credentials="/tmp/Z0ccUv/credentials" > --filter_gpu_resources="true" --framework_sorter="drf" --help="false" > --hostname_lookup="true" --http_authenticators="basic" > --http_framework_authenticators="basic" --initialize_driver_logging="true" > --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" > --max_agent_ping_timeouts="5" --max_completed_frameworks="50" > --max_completed_tasks_per_framework="1000" > --max_unreachable_tasks_per_framework="1000" > --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" > --registry="in_memory" --registry_fetch_timeout="1mins" > --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" > --registry_max_agent_count="102400" --registry_store_timeout="100secs" > --registry_strict="false" --root_submissions="true" --user_sorter="drf" > --version="false" --webui_dir="/mesos/mesos-1.4.0/_inst/share/mesos/webui" > --work_dir="/tmp/Z0ccUv/master" --zk_session_timeout="10secs" > I0817 01:00:06.827448 6734 master.cpp:494] Master only allowing > authenticated frameworks to register > I0817 01:00:06.827479 6734 master.cpp:508] Master only allowing > authenticated agents to register > I0817 01:00:06.827493 6734 master.cpp:521] Master only allowing > authenticated HTTP frameworks to register > I0817 01:00:06.827507 6734 credentials.hpp:37] Loading credentials for > authentication from '/tmp/Z0ccUv/credentials' > I0817 01:00:06.827826 6734 master.cpp:566] Using default 'crammd5' > authenticator > I0817 01:00:06.827980 6734 http.cpp:1026] Creating default 'basic' HTTP > authenticator for realm 'mesos-master-readonly' > I0817 01:00:06.828145 6734 http.cpp:1026] Creating default 'basic' HTTP > authenticator for realm 'mesos-master-readwrite' > I0817 01:00:06.828272 6734 http.cpp:1026] Creating default 'basic' HTTP > authenticator for realm 'mesos-master-scheduler' > I0817 01:00:06.828392 6734 master.cpp:646] Authorization enabled > I0817 01:00:06.828569 6743 hierarchical.cpp:171] Initialized hierarchical > allocator process > I0817 01:00:06.828590 6725 whitelist_watcher.cpp:77] No whitelist given > I0817 01:00:06.831037 6736 master.cpp:2163] Elected as the leading master! > I0817 01:00:06.831063 6736 master.cpp:1702] Recovering from registrar > I0817 01:00:06.831159 6728 registrar.cpp:347] Recovering registrar > I0817 01:00:06.831712 6728 registrar.cpp:391] Successfully fetched the > registry (0B) in 515840ns > I0817 01:00:06.831832 6728 registrar.cpp:495] Applied 1 operations in > 44686ns; attempting to update the registry > I0817 01:00:06.832372 6728 registrar.cpp:552] Successfully updated the > registry in 482048ns > I0817 01:00:06.832489 6728 registrar.cpp:424] Successfully recovered > registrar > I0817 01:00:06.832810 6738 master.cpp:1801] Recovered 0 agents from the > registry (129B); allowing 10mins for agents to re-register > I0817 01:00:06.832854 6744 hierarchical.cpp:209] Skipping recovery of > hierarchical allocator: nothing to recover > I0817 01:00:06.836031 6704 containerizer.cpp:246] Using isolation: > posix/cpu,posix/mem,filesystem/posix,network/cni,environment_secret > W0817 01:00:06.836477 6704 backend.cpp:76] Failed to create 'aufs' > backend: AufsBackend requires root privileges > W0817 01:00:06.836588 6704 backend.cpp:76] Failed to create 'bind' > backend: BindBackend requires root privileges > I0817 01:00:06.836623 6704 provisioner.cpp:255] Using default backend > 'copy' > I0817 01:00:06.840314 6704 process.cpp:3228] Attempting to spawn already > spawned process files@172.17.0.8:40444 > I0817 01:00:06.840575 6704 cluster.cpp:448] Creating default 'local' > authorizer > I0817 01:00:06.842278 6725 slave.cpp:250] Mesos agent started on (479)@ > 172.17.0.8:40444 > I0817 01:00:06.842310 6725 slave.cpp:251] Flags at startup: --acls="" > --appc_simple_discovery_uri_prefix="http://"; > --appc_store_dir="/tmp/SlaveRecoveryTest_0_RebootWithSlaveInfoMismatch_SEk3BV/store/appc" > --authenticate_http_executors="true" --authenticate_http_readonly="true" > --authenticate_http_readwrite="true" --authenticatee="crammd5" > --authentication_backoff_factor="1secs" --authorizer="local" > --cgroups_cpu_enable_pids_and_tids_count="false" > --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" > --cgroups_limit_swap="false" --cgroups_root="mesos" > --container_disk_watch_interval="15secs" --containerizers="mesos" > --credential="/tmp/SlaveRecoveryTest_0_RebootWithSlaveInfoMismatch_SEk3BV/credential" > --default_role="*" --disallow_sharing_agent_pid_namespace="false" > --disk_watch_interval="1mins" --docker="docker" > --docker_kill_orphans="true" --docker_registry="https://reg > istry-1.docker.io" --docker_remove_delay="6hrs" > --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" > --docker_store_dir="/tmp/SlaveRecoveryTest_0_RebootWithSlave > InfoMismatch_SEk3BV/store/docker" --docker_volume_checkpoint_dir > ="/var/run/mesos/isolators/docker/volume" > --enforce_container_disk_quota="false" > --executor_registration_timeout="1mins" > --executor_reregistration_timeout="2secs" > --executor_secret_key="/tmp/SlaveRecoveryTest_0_RebootWithSl > aveInfoMismatch_SEk3BV/executor_secret_key" > --executor_shutdown_grace_period="5secs" > --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_RebootWithSlaveInfoMismatch_SEk3BV/fetch" > --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" > --gc_disk_headroom="0.1" --hadoop_home="" --help="false" > --hostname_lookup="true" --http_command_executor="false" > --http_credentials="/tmp/SlaveRecoveryTest_0_RebootWithSlave > InfoMismatch_SEk3BV/http_credentials" --http_heartbeat_interval="30secs" > --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" > --launcher="posix" --launcher_dir="/mesos/mesos-1.4.0/_build/src" > --logbufsecs="0" --logging_level="INFO" > --max_completed_executors_per_framework="150" > --oversubscribed_resources_interval="15secs" --perf_duration="10secs" > --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" > --quiet="false" --recover="reconnect" --recovery_timeout="15mins" > --registration_backoff_factor="10ms" --resources="cpus:8;mem:4096;disk:2048" > --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecov > eryTest_0_RebootWithSlaveInfoMismatch_SEk3BV" > --sandbox_directory="/mnt/mesos/sandbox" --strict="true" > --switch_user="true" --systemd_enable_support="true" > --systemd_runtime_directory="/run/systemd/system" --version="false" > --work_dir="/tmp/SlaveRecoveryTest_0_RebootWithSlaveInfoMismatch_yF46Kd" > I0817 01:00:06.842685 6725 credentials.hpp:86] Loading credential for > authentication from '/tmp/SlaveRecoveryTest_0_Rebo > otWithSlaveInfoMismatch_SEk3BV/credential' > I0817 01:00:06.842784 6704 process.cpp:3228] Attempting to spawn already > spawned process version@172.17.0.8:40444 > I0817 01:00:06.842851 6725 slave.cpp:283] Agent using credential for: > test-principal > I0817 01:00:06.842875 6725 credentials.hpp:37] Loading credentials for > authentication from '/tmp/SlaveRecoveryTest_0_Rebo > otWithSlaveInfoMismatch_SEk3BV/http_credentials' > I0817 01:00:06.843179 6725 http.cpp:1026] Creating default 'basic' HTTP > authenticator for realm 'mesos-agent-executor' > I0817 01:00:06.843313 6725 http.cpp:1047] Creating default 'jwt' HTTP > authenticator for realm 'mesos-agent-executor' > I0817 01:00:06.843533 6725 http.cpp:1026] Creating default 'basic' HTTP > authenticator for realm 'mesos-agent-readonly' > I0817 01:00:06.843629 6725 http.cpp:1047] Creating default 'jwt' HTTP > authenticator for realm 'mesos-agent-readonly' > I0817 01:00:06.843639 6704 sched.cpp:232] Version: 1.4.0 > I0817 01:00:06.843857 6725 http.cpp:1026] Creating default 'basic' HTTP > authenticator for realm 'mesos-agent-readwrite' > I0817 01:00:06.843960 6725 http.cpp:1047] Creating default 'jwt' HTTP > authenticator for realm 'mesos-agent-readwrite' > I0817 01:00:06.844141 6731 sched.cpp:336] New master detected at > master@172.17.0.8:40444 > I0817 01:00:06.844230 6731 sched.cpp:407] Authenticating with master > master@172.17.0.8:40444 > I0817 01:00:06.844249 6731 sched.cpp:414] Using default CRAM-MD5 > authenticatee > I0817 01:00:06.844457 6738 authenticatee.cpp:121] Creating new client > SASL connection > I0817 01:00:06.844712 6736 master.cpp:7837] Authenticating > scheduler-db03a645-3ff8-4b33-b9cf-8617bdc60875@172.17.0.8:40444 > I0817 01:00:06.844805 6739 authenticator.cpp:414] Starting authentication > session for crammd5-authenticatee(994)@172.17.0.8:40444 > I0817 01:00:06.845022 6742 authenticator.cpp:98] Creating new server SASL > connection > I0817 01:00:06.845257 6729 authenticatee.cpp:213] Received SASL > authentication mechanisms: CRAM-MD5 > I0817 01:00:06.845285 6729 authenticatee.cpp:239] Attempting to > authenticate with mechanism 'CRAM-MD5' > I0817 01:00:06.845379 6730 authenticator.cpp:204] Received SASL > authentication start > I0817 01:00:06.845227 6725 slave.cpp:565] Agent resources: > [{"name":"cpus","scalar":{"value":8.0},"type":"SCALAR"},{" > name":"mem","scalar":{"value":4096.0},"type":"SCALAR"},{"nam > e":"disk","scalar":{"value":2048.0},"type":"SCALAR"},{"nam > e":"ports","ranges":{"range":[{"begin":31000,"end":32000}]}, > "type":"RANGES"}] > I0817 01:00:06.845443 6730 authenticator.cpp:326] Authentication requires > more steps > I0817 01:00:06.845474 6725 slave.cpp:573] Agent attributes: [ ] > I0817 01:00:06.845495 6725 slave.cpp:582] Agent hostname: 45197c1facb6 > I0817 01:00:06.845587 6726 authenticatee.cpp:259] Received SASL > authentication step > I0817 01:00:06.845631 6737 status_update_manager.cpp:177] Pausing sending > status updates > I0817 01:00:06.845721 6727 authenticator.cpp:232] Received SASL > authentication step > I0817 01:00:06.845752 6727 auxprop.cpp:109] Request to lookup properties > for user: 'test-principal' realm: '45197c1facb6' server FQDN: > '45197c1facb6' SASL_AUXPROP_VERIFY_AGAINST_HASH: false > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false > I0817 01:00:06.845765 6727 auxprop.cpp:181] Looking up auxiliary property > '*userPassword' > I0817 01:00:06.845803 6727 auxprop.cpp:181] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I0817 01:00:06.845824 6727 auxprop.cpp:109] Request to lookup properties > for user: 'test-principal' realm: '45197c1facb6' server FQDN: > '45197c1facb6' SASL_AUXPROP_VERIFY_AGAINST_HASH: false > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true > I0817 01:00:06.845834 6727 auxprop.cpp:131] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I0817 01:00:06.845840 6727 auxprop.cpp:131] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I0817 01:00:06.845859 6727 authenticator.cpp:318] Authentication success > I0817 01:00:06.845944 6722 authenticatee.cpp:299] Authentication success > I0817 01:00:06.846010 6733 master.cpp:7867] Successfully authenticated > principal 'test-principal' at scheduler-db03a645-3ff8-4b33-b > 9cf-8617bdc60875@172.17.0.8:40444 > I0817 01:00:06.846038 6732 authenticator.cpp:432] Authentication session > cleanup for crammd5-authenticatee(994)@172.17.0.8:40444 > I0817 01:00:06.846205 6724 sched.cpp:513] Successfully authenticated with > master master@172.17.0.8:40444 > I0817 01:00:06.846230 6724 sched.cpp:836] Sending SUBSCRIBE call to > master@172.17.0.8:40444 > I0817 01:00:06.846350 6724 sched.cpp:869] Will retry registration in > 1.760433846secs if necessary > I0817 01:00:06.846515 6735 master.cpp:2894] Received SUBSCRIBE call for > framework 'default' at scheduler-db03a645-3ff8-4b33-b > 9cf-8617bdc60875@172.17.0.8:40444 > terminate called after throwing an instance of 'std::system_error' > what(): Invalid argumentI0817 01:00:06.846544 6735 master.cpp:2228] > Authorizing framework principal 'test-principal' to receive offers for > roles '{ * }' > > *** Aborted at 1502931606 (unix time) try "date -d @1502931606" if you are > using GNU date *** > PC: @ 0x2b5638257c37 (unknown) > *** SIGABRT (@0x3e800001a30) received by PID 6704 (TID 0x2b56419b3700) > from PID 6704; stack trace: *** > I0817 01:00:06.847018 6744 master.cpp:2974] Subscribing framework default > with checkpointing enabled and capabilities [ RESERVATION_REFINEMENT ] > @ 0x2b5638013330 (unknown) > I0817 01:00:06.847276 6723 state.cpp:64] Recovering state from > '/tmp/SlaveRecoveryTest_0_RebootWithSlaveInfoMismatch_yF46Kd/meta' > @ 0x2b5638257c37 (unknown) > @ 0x2b563825b028 (unknown) > @ 0x2b5637843535 (unknown) > I0817 01:00:06.847627 6730 status_update_manager.cpp:203] Recovering > status update manager > I0817 01:00:06.847650 6742 sched.cpp:759] Framework registered with > 66249101-3204-4bec-9632-4a28a9d4b6e5-0000 > @ 0x2b56378416d6 (unknown) > I0817 01:00:06.847724 6742 sched.cpp:773] Scheduler::registered took > 38692ns > @ 0x2b5637841703 (unknown) > I0817 01:00:06.847841 6728 containerizer.cpp:609] Recovering containerizer > I0817 01:00:06.847849 6736 hierarchical.cpp:303] Added framework > 66249101-3204-4bec-9632-4a28a9d4b6e5-0000 > @ 0x2b5637894af5 (unknown) > I0817 01:00:06.848007 6736 hierarchical.cpp:1943] No allocations performed > I0817 01:00:06.848045 6736 hierarchical.cpp:2033] No inverse offers to > send out! > I0817 01:00:06.848076 6736 hierarchical.cpp:1486] Performed allocation > for 0 agents in 112255ns > @ 0x2b563800b184 start_thread > @ 0x2b563831effd (unknown) > make[4]: *** [check-local] Aborted (core dumped) > make[4]: Leaving directory `/mesos/mesos-1.4.0/_build/src' > make[3]: *** [check-am] Error 2 > make[3]: Leaving directory `/mesos/mesos-1.4.0/_build/src' > make[2]: *** [check] Error 2 > make[2]: Leaving directory `/mesos/mesos-1.4.0/_build/src' > make[1]: *** [check-recursive] Error 1 > make[1]: Leaving directory `/mesos/mesos-1.4.0/_build' > make: *** [distcheck] Error 1 > + docker rmi mesos-1502928841-11353 > Untagged: mesos-1502928841-11353:latest > Deleted: sha256:b357f94c675052121faba6fe659e4eb7ae3e1561b4a9437f6f1fa > 9372bd553f0 > Deleted: sha256:a96273e4fde97cd27620621e0595131961cc9d52b7596f71f0f52 > 27fe8fc13ca > Deleted: sha256:ce40dc29c56558c1c6bc6de09555cf72ac0ed0e123528596b9f00 > 366d0a5d9cb > Deleted: sha256:07c588a98d87917d607d4bfc5ed8e2aa505ca3ef14b805f03d6e2 > da329eb1bf8 > Deleted: sha256:db572215bdd853fdae5e743968629a440977c43ce9b85e7dd367a > 64703bca758 > Deleted: sha256:c53b35da4b823af8922b02331a25daa3929de567db25a5b768a9f > d878f9a6bdd > Deleted: sha256:a5e5c1c3346f346f44235f537f35bf5cd17ec098e7ff02ebc24a5 > c638ea1bfc8 > Deleted: sha256:926e91e72e8e57be634ea00e9ddff4e4cabbc11335a3fb219cd13 > 1b2ed9ab392 > Deleted: sha256:f4649fa91e399abcaff7e3ae53ff38d5e76fa41598cd59c2ad8ad > e0ab6733e4d > Deleted: sha256:743e6ebb05fbd1e05c86275466c6188b9fd4e45b8d50d3a1bd7bd > 72f3da6cf5c > Deleted: sha256:0692e3b9d64f5aa3b030779e4f9a76224e412d59a542ccbfd09e2 > 5fa6affb46f > Deleted: sha256:f28c43c772395095e3d7cbcb1f8e03e019c81e9180460f9d6e87b > c8c4222002a > Deleted: sha256:c885dcb5d6d14a85d75c216e122bea52342e8db9e83c8725d23a4 > c0a7c07c2b6 > Deleted: sha256:66500f988c2ab193d11a154febd6bd8cb1a13f9e270ea6cf0dc80 > bbfe1114bc6 > Deleted: sha256:f9a400d8c7f70691776f92bec8c4a15b3ed3e3beb5fdf2c77fa39 > d53d6ff4ce6 > Deleted: sha256:6b9a61dcfa10915fa155e3596d04544a0263a294f74d5720698fa > 3ce14cc89fa > Deleted: sha256:6c31420e1acbb33ca3a6a1a8fad9ee67ccdf4c51420ad66bb167c > fb879bf18d2 > Deleted: sha256:457d8b02047822f8e50c72c0f0077ae09497b3f30506b0ac2d6b8 > e8207d0f6bd > Deleted: sha256:1045dd5e6093bfae7f668960f2473b540a3c7e57305beb8613e97 > e74ea1786b7 > Deleted: sha256:b06f149fef2d09a542a4708153903f753990e4c608434a65d6b59 > 9ee4fed4bd9 > Deleted: sha256:0d51e647adbe81d4cb44a35a96734c9cd4ecff41a57ef91eac8f4 > 348fd546a12 > Deleted: sha256:63e2e1e2bf046565d1723dfbd4754867ed745e1feca4ceef5ba5b > 4c3d2d0b747 > Deleted: sha256:c2be339c2a2099eab71b3adf3c70c9b0e8ef5a615229504dfdcfa > a68bd6a92cf > Deleted: sha256:0e0dd36544fec166584cffc3867896fd04b9b733e55e7e7a30084 > a35d0ca5fd5 > Build step 'Execute shell' marked build as failure >
