https://issues.apache.org/jira/browse/MESOS-6544 https://issues.apache.org/jira/browse/MESOS-6545
On Thu, Nov 3, 2016 at 4:56 PM, Benjamin Mahler <bmah...@apache.org> wrote: > This test uses two mock executors across two agents, which means the > agents may concurrently call into the TestContainerizer to launch the > container. The TestContainerizer is not a Process and so we may > concurrently modify its data structures, I suspect that the following > executes concurrently and leads to the crash: > > https://github.com/apache/mesos/blob/44242c058158727ce013bd51764368 > f5e120ee75/src/tests/containerizer.cpp#L131 > > On Wed, Nov 2, 2016 at 6:41 PM, Apache Jenkins Server < > jenk...@builds.apache.org> wrote: > >> See <https://builds.apache.org/job/Mesos/BUILDTOOL=autotools,COM >> PILER=clang,CONFIGURATION=--verbose%20--enable-libevent%20 >> --enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubunt >> u%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&( >> !ubuntu-6)/2873/changes> >> >> Changes: >> >> [josephwu] Fixed some markdown whitespace style. >> >> [josephwu] Fixed a typo in `gpu-support.md`. >> >> [josephwu] Fixed FD leak due to exiting early in Mesos fetcher. >> >> ------------------------------------------ >> [...truncated 78174 lines...] >> I1103 01:40:55.530350 29098 slave.cpp:974] Authenticating with master >> master@172.17.0.2:58302 >> I1103 01:40:55.530432 29098 slave.cpp:985] Using default CRAM-MD5 >> authenticatee >> I1103 01:40:55.530627 29098 slave.cpp:947] Detecting new master >> I1103 01:40:55.530675 29108 authenticatee.cpp:121] Creating new client >> SASL connection >> I1103 01:40:55.530743 29098 slave.cpp:5587] Received oversubscribable >> resources {} from the resource estimator >> I1103 01:40:55.530961 29099 master.cpp:6742] Authenticating slave(150)@ >> 172.17.0.2:58302 >> I1103 01:40:55.531070 29112 authenticator.cpp:414] Starting >> authentication session for crammd5-authenticatee(357)@172.17.0.2:58302 >> I1103 01:40:55.531328 29106 authenticator.cpp:98] Creating new server >> SASL connection >> I1103 01:40:55.531561 29108 authenticatee.cpp:213] Received SASL >> authentication mechanisms: CRAM-MD5 >> I1103 01:40:55.531604 29108 authenticatee.cpp:239] Attempting to >> authenticate with mechanism 'CRAM-MD5' >> I1103 01:40:55.531713 29101 authenticator.cpp:204] Received SASL >> authentication start >> I1103 01:40:55.531805 29101 authenticator.cpp:326] Authentication >> requires more steps >> I1103 01:40:55.531921 29108 authenticatee.cpp:259] Received SASL >> authentication step >> I1103 01:40:55.532120 29101 authenticator.cpp:232] Received SASL >> authentication step >> I1103 01:40:55.532155 29101 auxprop.cpp:109] Request to lookup properties >> for user: 'test-principal' realm: '3a1c598ce334' server FQDN: >> '3a1c598ce334' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false >> I1103 01:40:55.532179 29101 auxprop.cpp:181] Looking up auxiliary >> property '*userPassword' >> I1103 01:40:55.532233 29101 auxprop.cpp:181] Looking up auxiliary >> property '*cmusaslsecretCRAM-MD5' >> I1103 01:40:55.532266 29101 auxprop.cpp:109] Request to lookup properties >> for user: 'test-principal' realm: '3a1c598ce334' server FQDN: >> '3a1c598ce334' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true >> I1103 01:40:55.532289 29101 auxprop.cpp:131] Skipping auxiliary property >> '*userPassword' since SASL_AUXPROP_AUTHZID == true >> I1103 01:40:55.532305 29101 auxprop.cpp:131] Skipping auxiliary property >> '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true >> I1103 01:40:55.532335 29101 authenticator.cpp:318] Authentication success >> I1103 01:40:55.532413 29110 authenticatee.cpp:299] Authentication success >> I1103 01:40:55.532467 29108 master.cpp:6772] Successfully authenticated >> principal 'test-principal' at slave(150)@172.17.0.2:58302 >> I1103 01:40:55.532536 29111 authenticator.cpp:432] Authentication session >> cleanup for crammd5-authenticatee(357)@172.17.0.2:58302 >> I1103 01:40:55.532755 29098 slave.cpp:1069] Successfully authenticated >> with master master@172.17.0.2:58302 >> I1103 01:40:55.532997 29098 slave.cpp:1483] Will retry registration in >> 12.590371ms if necessary >> I1103 01:40:55.533179 29108 master.cpp:5151] Registering agent at >> slave(150)@172.17.0.2:58302 (maintenance-host-2) with id >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 >> I1103 01:40:55.533572 29112 registrar.cpp:461] Applied 1 operations in >> 94467ns; attempting to update the registry >> I1103 01:40:55.546341 29107 slave.cpp:1483] Will retry registration in >> 36.501523ms if necessary >> I1103 01:40:55.546461 29099 master.cpp:5139] Ignoring register agent >> message from slave(150)@172.17.0.2:58302 (maintenance-host-2) as >> admission is already in progress >> I1103 01:40:55.565403 29097 leveldb.cpp:341] Persisting action (16 bytes) >> to leveldb took 48.099208ms >> I1103 01:40:55.565495 29097 replica.cpp:708] Persisted action TRUNCATE at >> position 4 >> I1103 01:40:55.566788 29097 replica.cpp:691] Replica received learned >> notice for position 4 from @0.0.0.0:0 >> I1103 01:40:55.583937 29101 slave.cpp:1483] Will retry registration in >> 26.127711ms if necessary >> I1103 01:40:55.584123 29112 master.cpp:5139] Ignoring register agent >> message from slave(150)@172.17.0.2:58302 (maintenance-host-2) as >> admission is already in progress >> I1103 01:40:55.609695 29097 leveldb.cpp:341] Persisting action (18 bytes) >> to leveldb took 42.905697ms >> I1103 01:40:55.609860 29097 leveldb.cpp:399] Deleting ~2 keys from >> leveldb took 96623ns >> I1103 01:40:55.609899 29097 replica.cpp:708] Persisted action TRUNCATE at >> position 4 >> I1103 01:40:55.611063 29106 log.cpp:577] Attempting to append 513 bytes >> to the log >> I1103 01:40:55.611229 29097 coordinator.cpp:348] Coordinator attempting >> to write APPEND action at position 5 >> I1103 01:40:55.611498 29100 slave.cpp:1483] Will retry registration in >> 85.55417ms if necessary >> I1103 01:40:55.612069 29105 master.cpp:5139] Ignoring register agent >> message from slave(150)@172.17.0.2:58302 (maintenance-host-2) as >> admission is already in progress >> I1103 01:40:55.612313 29100 replica.cpp:537] Replica received write >> request for position 5 from __req_res__(2518)@172.17.0.2:58302 >> I1103 01:40:55.657845 29100 leveldb.cpp:341] Persisting action (532 >> bytes) to leveldb took 45.517822ms >> I1103 01:40:55.657938 29100 replica.cpp:708] Persisted action APPEND at >> position 5 >> I1103 01:40:55.658681 29103 replica.cpp:691] Replica received learned >> notice for position 5 from @0.0.0.0:0 >> I1103 01:40:55.698050 29112 slave.cpp:1483] Will retry registration in >> 100.32384ms if necessary >> I1103 01:40:55.698319 29101 master.cpp:5139] Ignoring register agent >> message from slave(150)@172.17.0.2:58302 (maintenance-host-2) as >> admission is already in progress >> I1103 01:40:55.705086 29103 leveldb.cpp:341] Persisting action (534 >> bytes) to leveldb took 46.384557ms >> I1103 01:40:55.705157 29103 replica.cpp:708] Persisted action APPEND at >> position 5 >> I1103 01:40:55.707480 29098 registrar.cpp:506] Successfully updated the >> registry in 173824us >> I1103 01:40:55.707741 29100 log.cpp:596] Attempting to truncate the log >> to 5 >> I1103 01:40:55.708029 29110 coordinator.cpp:348] Coordinator attempting >> to write TRUNCATE action at position 6 >> I1103 01:40:55.708501 29097 slave.cpp:4251] Received ping from >> slave-observer(151)@172.17.0.2:58302 >> I1103 01:40:55.708528 29112 master.cpp:5222] Registered agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 at slave(150)@172.17.0.2:58302 >> (maintenance-host-2) with cpus(*):2; mem(*):1024; disk(*):1024; >> ports(*):[31000-32000] >> I1103 01:40:55.708796 29097 slave.cpp:1115] Registered with master >> master@172.17.0.2:58302; given agent ID 3167a687-904b-4b57-bc0f-91b67d >> c7e41d-S1 >> I1103 01:40:55.708822 29097 fetcher.cpp:86] Clearing fetcher cache >> I1103 01:40:55.708889 29103 replica.cpp:537] Replica received write >> request for position 6 from __req_res__(2519)@172.17.0.2:58302 >> I1103 01:40:55.708935 29100 hierarchical.cpp:485] Added agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 (maintenance-host-2) with >> cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: {}) >> I1103 01:40:55.709065 29105 status_update_manager.cpp:184] Resuming >> sending status updates >> I1103 01:40:55.709139 29100 hierarchical.cpp:1694] No allocations >> performed >> I1103 01:40:55.709204 29100 hierarchical.cpp:1309] Performed allocation >> for agent 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 in 229932ns >> I1103 01:40:55.709316 29097 slave.cpp:1138] Checkpointing SlaveInfo to >> '/tmp/MasterMaintenanceTest_InverseOffersFilters_xHOK8S/meta >> /slaves/3167a687-904b-4b57-bc0f-91b67dc7e41d-S1/slave.info' >> I1103 01:40:55.709650 29097 slave.cpp:1175] Forwarding total >> oversubscribed resources {} >> I1103 01:40:55.709787 29097 master.cpp:5621] Received update of agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 at slave(150)@172.17.0.2:58302 >> (maintenance-host-2) with total oversubscribed resources {} >> I1103 01:40:55.710000 29108 hierarchical.cpp:555] Agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 (maintenance-host-2) updated >> with oversubscribed resources {} (total: cpus(*):2; mem(*):1024; >> disk(*):1024; ports(*):[31000-32000], allocated: {}) >> I1103 01:40:55.710144 29108 hierarchical.cpp:1694] No allocations >> performed >> I1103 01:40:55.710199 29108 hierarchical.cpp:1309] Performed allocation >> for agent 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 in 157491ns >> I1103 01:40:55.711632 29105 process.cpp:3570] Handling HTTP event for >> process 'master' with path: '/master/maintenance/schedule' >> I1103 01:40:55.712441 29111 http.cpp:391] HTTP POST for >> /master/maintenance/schedule from 172.17.0.2:45755 >> I1103 01:40:55.713325 29110 registrar.cpp:461] Applied 1 operations in >> 165622ns; attempting to update the registry >> I1103 01:40:55.756808 29103 leveldb.cpp:341] Persisting action (16 bytes) >> to leveldb took 47.899715ms >> I1103 01:40:55.756888 29103 replica.cpp:708] Persisted action TRUNCATE at >> position 6 >> I1103 01:40:55.758083 29103 replica.cpp:691] Replica received learned >> notice for position 6 from @0.0.0.0:0 >> I1103 01:40:55.807322 29103 leveldb.cpp:341] Persisting action (18 bytes) >> to leveldb took 49.23844ms >> I1103 01:40:55.807481 29103 leveldb.cpp:399] Deleting ~2 keys from >> leveldb took 90240ns >> I1103 01:40:55.807517 29103 replica.cpp:708] Persisted action TRUNCATE at >> position 6 >> I1103 01:40:55.808619 29101 log.cpp:577] Attempting to append 732 bytes >> to the log >> I1103 01:40:55.808737 29097 coordinator.cpp:348] Coordinator attempting >> to write APPEND action at position 7 >> I1103 01:40:55.809571 29109 replica.cpp:537] Replica received write >> request for position 7 from __req_res__(2520)@172.17.0.2:58302 >> I1103 01:40:55.810784 29098 hierarchical.cpp:1694] No allocations >> performed >> I1103 01:40:55.810860 29098 hierarchical.cpp:1286] Performed allocation >> for 2 agents in 282405ns >> I1103 01:40:55.870082 29109 leveldb.cpp:341] Persisting action (751 >> bytes) to leveldb took 60.489282ms >> I1103 01:40:55.870177 29109 replica.cpp:708] Persisted action APPEND at >> position 7 >> I1103 01:40:55.871421 29109 replica.cpp:691] Replica received learned >> notice for position 7 from @0.0.0.0:0 >> I1103 01:40:55.976619 29109 leveldb.cpp:341] Persisting action (753 >> bytes) to leveldb took 105.201942ms >> I1103 01:40:55.976718 29109 replica.cpp:708] Persisted action APPEND at >> position 7 >> I1103 01:40:55.979708 29109 registrar.cpp:506] Successfully updated the >> registry in 266.301952ms >> I1103 01:40:55.979878 29103 log.cpp:596] Attempting to truncate the log >> to 7 >> I1103 01:40:55.979991 29106 coordinator.cpp:348] Coordinator attempting >> to write TRUNCATE action at position 8 >> I1103 01:40:55.980255 29112 master.cpp:5683] Updating unavailability of >> agent 3167a687-904b-4b57-bc0f-91b67dc7e41d-S0 at slave(149)@ >> 172.17.0.2:58302 (maintenance-host), starting at 2449.0107815972weeks >> I1103 01:40:55.980481 29112 master.cpp:5683] Updating unavailability of >> agent 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 at slave(150)@ >> 172.17.0.2:58302 (maintenance-host-2), starting at 2449.0107815972weeks >> I1103 01:40:55.980818 29097 replica.cpp:537] Replica received write >> request for position 8 from __req_res__(2521)@172.17.0.2:58302 >> I1103 01:40:55.981092 29103 hierarchical.cpp:1694] No allocations >> performed >> I1103 01:40:55.981150 29103 hierarchical.cpp:1309] Performed allocation >> for agent 3167a687-904b-4b57-bc0f-91b67dc7e41d-S0 in 199516ns >> I1103 01:40:55.981317 29103 hierarchical.cpp:1694] No allocations >> performed >> I1103 01:40:55.981367 29103 hierarchical.cpp:1309] Performed allocation >> for agent 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 in 144365ns >> I1103 01:40:55.982864 29079 scheduler.cpp:176] Version: 1.2.0 >> I1103 01:40:55.983412 29112 scheduler.cpp:469] New master detected at >> master@172.17.0.2:58302 >> I1103 01:40:55.983438 29112 scheduler.cpp:478] Waiting for 0ns before >> initiating a re-(connection) attempt with the master >> I1103 01:40:55.984688 29098 scheduler.cpp:353] Connected with the master >> at http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> >> 01:40:55.985872 29109 scheduler.cpp:235] Sending SUBSCRIBE call to >> http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> >> 01:40:55.986587 29099 process.cpp:3570] Handling HTTP event for process >> 'master' with path: '/master/api/v1/scheduler' >> I1103 01:40:55.987659 29108 http.cpp:391] HTTP POST for >> /master/api/v1/scheduler from 172.17.0.2:45771 >> I1103 01:40:55.988019 29108 master.cpp:2329] Received subscription >> request for HTTP framework 'default' >> I1103 01:40:55.988088 29108 master.cpp:2069] Authorizing framework >> principal 'test-principal' to receive offers for role '*' >> I1103 01:40:55.988632 29099 master.cpp:2427] Subscribing framework >> 'default' with checkpointing disabled and capabilities [ ] >> I1103 01:40:55.989305 29105 hierarchical.cpp:275] Added framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:55.989459 29099 master.hpp:2161] Sending heartbeat to >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:55.990151 29108 scheduler.cpp:675] Enqueuing event SUBSCRIBED >> received from http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> >> 01:40:55.990648 29108 scheduler.cpp:675] Enqueuing event HEARTBEAT received >> from http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> >> 01:40:55.991364 29105 hierarchical.cpp:1286] Performed allocation for 2 >> agents in 2.044414ms >> I1103 01:40:55.992657 29100 master.cpp:6571] Sending 2 offers to >> framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 (default) >> I1103 01:40:55.994067 29100 master.cpp:6661] Sending 2 inverse offers to >> framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 (default) >> I1103 01:40:55.995620 29101 scheduler.cpp:675] Enqueuing event OFFERS >> received from http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> >> 01:40:55.996536 29101 scheduler.cpp:675] Enqueuing event INVERSE_OFFERS >> received from http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> >> 01:40:56.060276 29097 leveldb.cpp:341] Persisting action (16 bytes) to >> leveldb took 79.452553ms >> I1103 01:40:56.060681 29097 replica.cpp:708] Persisted action TRUNCATE at >> position 8 >> I1103 01:40:56.061957 29102 replica.cpp:691] Replica received learned >> notice for position 8 from @0.0.0.0:0 >> I1103 01:40:56.118927 29102 leveldb.cpp:341] Persisting action (18 bytes) >> to leveldb took 56.925572ms >> I1103 01:40:56.119022 29102 leveldb.cpp:399] Deleting ~2 keys from >> leveldb took 51598ns >> I1103 01:40:56.119046 29102 replica.cpp:708] Persisted action TRUNCATE at >> position 8 >> I1103 01:40:56.120704 29099 hierarchical.cpp:1694] No allocations >> performed >> I1103 01:40:56.121099 29099 hierarchical.cpp:1286] Performed allocation >> for 2 agents in 610047ns >> I1103 01:40:56.122525 29097 scheduler.cpp:235] Sending ACCEPT call to >> http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> 01:40:56.123138 >> 29097 scheduler.cpp:235] Sending ACCEPT call to >> http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> >> 01:40:56.124460 29098 process.cpp:3570] Handling HTTP event for process >> 'master' with path: '/master/api/v1/scheduler' >> I1103 01:40:56.124822 29098 process.cpp:3570] Handling HTTP event for >> process 'master' with path: '/master/api/v1/scheduler' >> I1103 01:40:56.126590 29098 http.cpp:391] HTTP POST for >> /master/api/v1/scheduler from 172.17.0.2:45772 >> I1103 01:40:56.127682 29098 master.cpp:3581] Processing ACCEPT call for >> offers: [ 3167a687-904b-4b57-bc0f-91b67dc7e41d-O0 ] on agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S0 at slave(149)@172.17.0.2:58302 >> (maintenance-host) for framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> (default) >> I1103 01:40:56.127856 29098 master.cpp:3173] Authorizing framework >> principal 'test-principal' to launch task 19b41cf9-ba0f-472e-8cae-5ea705 >> 74b1b4 >> I1103 01:40:56.128916 29098 http.cpp:391] HTTP POST for >> /master/api/v1/scheduler from 172.17.0.2:45772 >> I1103 01:40:56.129622 29098 master.cpp:3581] Processing ACCEPT call for >> offers: [ 3167a687-904b-4b57-bc0f-91b67dc7e41d-O1 ] on agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 at slave(150)@172.17.0.2:58302 >> (maintenance-host-2) for framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> (default) >> I1103 01:40:56.129730 29098 master.cpp:3173] Authorizing framework >> principal 'test-principal' to launch task 846a812b-b562-4782-b0fd-9e760a >> 52306e >> W1103 01:40:56.131443 29098 validation.cpp:920] Executor 'executor-1' for >> task '19b41cf9-ba0f-472e-8cae-5ea70574b1b4' uses less CPUs (None) than >> the minimum required (0.01). Please update your executor, as this will be >> mandatory in future releases. >> W1103 01:40:56.131489 29098 validation.cpp:932] Executor 'executor-1' for >> task '19b41cf9-ba0f-472e-8cae-5ea70574b1b4' uses less memory (None) than >> the minimum required (32MB). Please update your executor, as this will be >> mandatory in future releases. >> I1103 01:40:56.132071 29098 master.cpp:8334] Adding task >> 19b41cf9-ba0f-472e-8cae-5ea70574b1b4 with resources cpus(*):2; >> mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S0 (maintenance-host) >> I1103 01:40:56.132442 29098 master.cpp:4230] Launching task >> 19b41cf9-ba0f-472e-8cae-5ea70574b1b4 of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 (default) with resources >> cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S0 at slave(149)@172.17.0.2:58302 >> (maintenance-host) >> W1103 01:40:56.133824 29098 validation.cpp:920] Executor 'executor-2' for >> task '846a812b-b562-4782-b0fd-9e760a52306e' uses less CPUs (None) than >> the minimum required (0.01). Please update your executor, as this will be >> mandatory in future releases. >> W1103 01:40:56.133858 29098 validation.cpp:932] Executor 'executor-2' for >> task '846a812b-b562-4782-b0fd-9e760a52306e' uses less memory (None) than >> the minimum required (32MB). Please update your executor, as this will be >> mandatory in future releases. >> I1103 01:40:56.134299 29097 slave.cpp:1547] Got assigned task >> '19b41cf9-ba0f-472e-8cae-5ea70574b1b4' for framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.134332 29098 master.cpp:8334] Adding task >> 846a812b-b562-4782-b0fd-9e760a52306e with resources cpus(*):2; >> mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 (maintenance-host-2) >> I1103 01:40:56.134572 29098 master.cpp:4230] Launching task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 (default) with resources >> cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 at slave(150)@172.17.0.2:58302 >> (maintenance-host-2) >> I1103 01:40:56.135084 29097 slave.cpp:1709] Launching task >> '19b41cf9-ba0f-472e-8cae-5ea70574b1b4' for framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.135145 29098 slave.cpp:1547] Got assigned task >> '846a812b-b562-4782-b0fd-9e760a52306e' for framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.135660 29098 slave.cpp:1709] Launching task >> '846a812b-b562-4782-b0fd-9e760a52306e' for framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.135871 29097 paths.cpp:536] Trying to chown >> '/tmp/MasterMaintenanceTest_InverseOffersFilters_B4KX1H/slav >> es/3167a687-904b-4b57-bc0f-91b67dc7e41d-S0/frameworks/ >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000/executors/ >> executor-1/runs/fbb81e53-b134-4196-9385-791612794545' to user 'mesos' >> I1103 01:40:56.138550 29098 paths.cpp:536] Trying to chown >> '/tmp/MasterMaintenanceTest_InverseOffersFilters_xHOK8S/slav >> es/3167a687-904b-4b57-bc0f-91b67dc7e41d-S1/frameworks/ >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000/executors/ >> executor-2/runs/c7d8cc2e-b173-4dbe-b5ae-e45402c1cdf4' to user 'mesos' >> I1103 01:40:56.144282 29098 slave.cpp:6307] Launching executor >> 'executor-2' of framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 with >> resources {} in work directory '/tmp/MasterMaintenanceTest_In >> verseOffersFilters_xHOK8S/slaves/3167a687-904b-4b57-bc0f- >> 91b67dc7e41d-S1/frameworks/3167a687-904b-4b57-bc0f- >> 91b67dc7e41d-0000/executors/executor-2/runs/c7d8cc2e-b173- >> 4dbe-b5ae-e45402c1cdf4' >> I1103 01:40:56.144282 29097 slave.cpp:6307] Launching executor >> 'executor-1' of framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 with >> resources {} in work directory '/tmp/MasterMaintenanceTest_In >> verseOffersFilters_B4KX1H/slaves/3167a687-904b-4b57-bc0f- >> 91b67dc7e41d-S0/frameworks/3167a687-904b-4b57-bc0f- >> 91b67dc7e41d-0000/executors/executor-1/runs/fbb81e53-b134- >> 4196-9385-791612794545' >> F1103 01:40:56.144891 29097 owned.hpp:110] Check failed: 'get()' Must be >> non NULL >> *** Check failure stack trace: *** >> I1103 01:40:56.147233 29098 exec.cpp:162] Version: 1.2.0 >> I1103 01:40:56.147598 29112 exec.cpp:212] Executor started at: >> executor(60)@172.17.0.2:58302 with pid 29079 >> I1103 01:40:56.147956 29098 slave.cpp:2031] Queued task >> '846a812b-b562-4782-b0fd-9e760a52306e' for executor 'executor-2' of >> framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.148043 29098 slave.cpp:868] Successfully attached file >> '/tmp/MasterMaintenanceTest_InverseOffersFilters_xHOK8S/slav >> es/3167a687-904b-4b57-bc0f-91b67dc7e41d-S1/frameworks/ >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000/executors/ >> executor-2/runs/c7d8cc2e-b173-4dbe-b5ae-e45402c1cdf4' >> I1103 01:40:56.148277 29098 slave.cpp:3305] Got registration for executor >> 'executor-2' of framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 from >> executor(60)@172.17.0.2:58302 >> I1103 01:40:56.148833 29105 exec.cpp:237] Executor registered on agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 >> I1103 01:40:56.148895 29105 exec.cpp:249] Executor::registered took >> 29894ns >> I1103 01:40:56.149114 29098 slave.cpp:2247] Sending queued task >> '846a812b-b562-4782-b0fd-9e760a52306e' to executor 'executor-2' of >> framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 at executor(60)@ >> 172.17.0.2:58302 >> I1103 01:40:56.149495 29106 exec.cpp:324] Executor asked to run task >> '846a812b-b562-4782-b0fd-9e760a52306e' >> I1103 01:40:56.149608 29106 exec.cpp:333] Executor::launchTask took >> 98162ns >> I1103 01:40:56.149705 29106 exec.cpp:550] Executor sending status update >> TASK_RUNNING (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.149987 29106 slave.cpp:3740] Handling status update >> TASK_RUNNING (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 from executor(60)@ >> 172.17.0.2:58302 >> I1103 01:40:56.150545 29098 status_update_manager.cpp:323] Received >> status update TASK_RUNNING (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) >> for task 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.150606 29098 status_update_manager.cpp:500] Creating >> StatusUpdate stream for task 846a812b-b562-4782-b0fd-9e760a52306e of >> framework 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.151116 29098 status_update_manager.cpp:377] Forwarding >> update TASK_RUNNING (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) for >> task 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 to the agent >> @ 0x2b56852902fd google::LogMessage::Fail() >> I1103 01:40:56.151470 29101 slave.cpp:4169] Forwarding the update >> TASK_RUNNING (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 to master@172.17.0.2:58302 >> I1103 01:40:56.151757 29101 slave.cpp:4063] Status update manager >> successfully handled status update TASK_RUNNING (UUID: >> 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.151860 29101 slave.cpp:4079] Sending acknowledgement for >> status update TASK_RUNNING (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) >> for task 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 to executor(60)@ >> 172.17.0.2:58302 >> I1103 01:40:56.151954 29111 master.cpp:5757] Status update TASK_RUNNING >> (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 from agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 at slave(150)@172.17.0.2:58302 >> (maintenance-host-2) >> I1103 01:40:56.152041 29111 master.cpp:5819] Forwarding status update >> TASK_RUNNING (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.152123 29100 exec.cpp:373] Executor received status update >> acknowledgement 1fe05bba-4be1-4c4e-a847-e4e3289b2d26 for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.152546 29111 master.cpp:7712] Updating the state of task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 (latest state: TASK_RUNNING, >> status update state: TASK_RUNNING) >> I1103 01:40:56.153616 29103 scheduler.cpp:675] Enqueuing event UPDATE >> received from http://172.17.0.2:58302/master/api/v1/scheduler >> @ 0x2b568528f6de google::LogMessage::SendToLog() >> I1103 01:40:56.164335 29109 scheduler.cpp:235] Sending ACKNOWLEDGE call >> to http://172.17.0.2:58302/master/api/v1/scheduler >> I1103 <http://172.17.0.2:58302/master/api/v1/schedulerI1103> >> 01:40:56.165396 29109 process.cpp:3570] Handling HTTP event for process >> 'master' with path: '/master/api/v1/scheduler' >> I1103 01:40:56.167019 29109 http.cpp:391] HTTP POST for >> /master/api/v1/scheduler from 172.17.0.2:45772 >> I1103 01:40:56.167212 29109 master.cpp:4867] Processing ACKNOWLEDGE call >> 1fe05bba-4be1-4c4e-a847-e4e3289b2d26 for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 (default) on agent >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-S1 >> I1103 01:40:56.168145 29110 status_update_manager.cpp:395] Received >> status update acknowledgement (UUID: 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) >> for task 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> I1103 01:40:56.168666 29110 slave.cpp:3022] Status update manager >> successfully handled status update acknowledgement (UUID: >> 1fe05bba-4be1-4c4e-a847-e4e3289b2d26) for task >> 846a812b-b562-4782-b0fd-9e760a52306e of framework >> 3167a687-904b-4b57-bc0f-91b67dc7e41d-0000 >> @ 0x2b568528ffbd google::LogMessage::Flush() >> @ 0x2b5685293438 google::LogMessageFatal::~LogMessageFatal() >> @ 0xcb1fc4 google::CheckNotNull<>() >> @ 0xc93ba6 process::Owned<>::operator->() >> @ 0xc9100c mesos::internal::tests::TestCo >> ntainerizer::_launch() >> @ 0xc98182 _ZN7testing8internal12InvokeHe >> lperIN7process6FutureIbEESt5tupleIJRKN5mesos11ContainerIDERK >> 6OptionINS6_8TaskInfoEERKNS6_12ExecutorInfoERKSsRKSA_ISsERKN >> S6_7SlaveIDERKSt3mapISsSsSt4lessISsESaISt4pairISI_SsEEEbEEE1 >> 2InvokeMethodINS6_8internal5tests17TestContainerizerEMS14_ >> FS4_S9_SE_SH_SJ_SM_SP_SY_bEEES4_PT_T0_RKSZ_ >> @ 0xc98014 _ZNK7testing8internal18InvokeM >> ethodActionIN5mesos8internal5tests17TestContainerizerEMS5_FN >> 7process6FutureIbEERKNS2_11ContainerIDERK6OptionINS2_8TaskIn >> foEERKNS2_12ExecutorInfoERKSsRKSC_ISsERKNS2_7SlaveIDERKSt3ma >> pISsSsSt4lessISsESaISt4pairISK_SsEEEbEE7PerformIS8_St5tupleI >> JSB_SG_SJ_SL_SO_SR_S10_bEEEET_RKT0_ >> @ 0xc97fb6 _ZN7testing17PolymorphicAction >> INS_8internal18InvokeMethodActionIN5mesos8internal5tests17Te >> stContainerizerEMS6_FN7process6FutureIbEERKNS3_11ContainerID >> ERK6OptionINS3_8TaskInfoEERKNS3_12ExecutorInfoERKSsRKSD_ISsE >> RKNS3_7SlaveIDERKSt3mapISsSsSt4lessISsESaISt4pairISL_SsEEEbE >> EEE15MonomorphicImplIFS9_SC_SH_SK_SM_SP_SS_S11_bEE7Perform >> ERKSt5tupleIJSC_SH_SK_SM_SP_SS_S11_bEE >> @ 0xb51d58 _ZNK7testing6ActionIFN7process >> 6FutureIbEERKN5mesos11ContainerIDERK6OptionINS4_8TaskInfoEER >> KNS4_12ExecutorInfoERKSsRKS8_ISsERKNS4_7SlaveIDERKSt3mapISsS >> sSt4lessISsESaISt4pairISG_SsEEEbEE7PerformERKSt5tupleIJS7_ >> SC_SF_SH_SK_SN_SW_bEE >> @ 0xb51b6a testing::internal::ActionResul >> tHolder<>::PerformAction<>() >> @ 0xb4ca1c testing::internal::FunctionMoc >> kerBase<>::UntypedPerformAction() >> @ 0x1daa619 testing::internal::UntypedFunc >> tionMockerBase::UntypedInvokeWith() >> @ 0xb4995b _ZN7testing8internal18Function >> MockerBaseIFN7process6FutureIbEERKN5mesos11ContainerIDERK6Op >> tionINS5_8TaskInfoEERKNS5_12ExecutorInfoERKSsRKS9_ISsERKNS5_ >> 7SlaveIDERKSt3mapISsSsSt4lessISsESaISt4pairISH_SsEEEbEE10Inv >> okeWithERKSt5tupleIJS8_SD_SG_SI_SL_SO_SX_bEE >> @ 0xb49916 testing::internal::FunctionMocker<>::Invoke() >> @ 0xc9712f mesos::internal::tests::TestCo >> ntainerizer::launch() >> @ 0x2b5684298b9b mesos::internal::slave::Framew >> ork::launchExecutor() >> @ 0x2b568429607e mesos::internal::slave::Slave::_run() >> @ 0x2b56843265a0 _ZZN7process8dispatchIN5mesos8 >> internal5slave5SlaveERKNS_6FutureIbEERKNS1_13FrameworkInfoER >> KNS1_12ExecutorInfoERK6OptionINS1_8TaskInfoEERKSF_INS1_13Tas >> kGroupInfoEES6_S9_SC_SH_SL_EEvRKNS_3PIDIT_EEMSP_FvT0_T1_ >> T2_T3_T4_ET5_T6_T7_T8_T9_ENKUlPNS_11ProcessBaseEE_clES16_ >> @ 0x2b5684326062 _ZNSt17_Function_handlerIFvPN7 >> process11ProcessBaseEEZNS0_8dispatchIN5mesos8internal5slave5 >> SlaveERKNS0_6FutureIbEERKNS5_13FrameworkInfoERKNS5_12Executo >> rInfoERK6OptionINS5_8TaskInfoEERKSJ_INS5_13TaskGroupInfoEESA >> _SD_SG_SL_SP_EEvRKNS0_3PIDIT_EEMST_FvT0_T1_T2_T3_T4_ET5_T6_ >> T7_T8_T9_EUlS2_E_E9_M_invokeERKSt9_Any_dataS2_ >> @ 0x2b56851b2dd8 std::function<>::operator()() >> @ 0x2b5685199f74 process::ProcessBase::visit() >> @ 0x2b56851f84ee process::DispatchEvent::visit() >> @ 0x8dabe1 process::ProcessBase::serve() >> @ 0x2b5685197c64 process::ProcessManager::resume() >> @ 0x2b56851a28cc process::ProcessManager::init_ >> threads()::$_0::operator()() >> @ 0x2b56851a27d5 _ZNSt12_Bind_simpleIFZN7proces >> s14ProcessManager12init_threadsEvE3$_0vEE9_M_ >> invokeIJEEEvSt12_Index_tupleIJXspT_EEE >> @ 0x2b56851a27a5 std::_Bind_simple<>::operator()() >> @ 0x2b56851a277c std::thread::_Impl<>::_M_run() >> @ 0x2b5686d10a60 (unknown) >> @ 0x2b5687487184 start_thread >> make[4]: *** [check-local] Aborted >> make[4]: Leaving directory `/mesos/mesos-1.2.0/_build/src' >> make[3]: *** [check-am] Error 2 >> make[3]: Leaving directory `/mesos/mesos-1.2.0/_build/src' >> make[2]: *** [check] Error 2 >> make[2]: Leaving directory `/mesos/mesos-1.2.0/_build/src' >> make[1]: *** [check-recursive] Error 1 >> make[1]: Leaving directory `/mesos/mesos-1.2.0/_build' >> make: *** [distcheck] Error 1 >> + docker rmi mesos-1478134041-23345 >> Untagged: mesos-1478134041-23345:latest >> Deleted: sha256:74596d8e8b76315d1115980623d7e5af9150c10d54203c726895a >> a5cac6da79e >> Deleted: sha256:7a0b0a4cf4ac2c9897b51052cb68965f0b4e8a305ff6db16b7d53 >> 0709b6e406c >> Deleted: sha256:567f543714141c74573998d4209337287ffbe0e8ccd9ee8e4b044 >> cc6e872c0e5 >> Deleted: sha256:fbd8be03b0d3f1c70dea858388d951374325b608d3e56c6b51fd1 >> 02b193afed2 >> Deleted: sha256:7c7aee38843610c581d675c5620d8981f65d7fda18fc0fa05a264 >> fb510dd927c >> Deleted: sha256:654d69e9689193c66be28a0fff218d21a5a119fbeecdc26a8a4cf >> 34b3c2cd104 >> Deleted: sha256:4f5e0007d3903ea948c9fa13032fad2b1244dc0d4dbbee0673cd3 >> 470c8f3c24a >> Deleted: sha256:38239a48898a69f2bf752d732a101a22038fc2d8e023b6c789987 >> daa2332ee1b >> Deleted: sha256:329c7b62b50ac8578b3499838a1f12435013f0b4f429da2efac3d >> 76673800950 >> Deleted: sha256:45dc8a92ffad997c6e48598af8e4665ff06cbb6d6316351de7103 >> 17f420367a2 >> Deleted: sha256:bc4e9bad738f327a22d9bb1204cbb3787b6b04fcda850497d7a6f >> d54b1d9b9f3 >> Deleted: sha256:708772f854104da0e8b60eb0a9fb12c5cd85fca258ed74289e0ca >> dcab12bb2f9 >> Deleted: sha256:5498b1ca089ca60b79713f6a57bc63036a89079ae1afff2ec21d4 >> 42d5fa531f1 >> Deleted: sha256:7830805a402ac02a943fc9dbac0ff1dd46c4cfd67af2fb4e36b9d >> 2e43dd7a4d5 >> Deleted: sha256:354a09c0839db0533726726c863f23138aad1656c38ee8f3c16b0 >> 4b1766a0109 >> Deleted: sha256:73999f055e575a9767df52d22b67980d2b536b8ac1eeda0e9af1e >> a623fc939b9 >> Deleted: sha256:c915f730bb38299ca517a3f295fcdd21e0078d959381ae5a9aff0 >> f39b4748bd5 >> Deleted: sha256:ca688ce3b2900d35c52872f38fd4af8530b780bb362e6e1a42687 >> d4f5e4082e8 >> Deleted: sha256:d2c33d1bb8213d5ed68c888aa09a65370a36d6ab4311704d2f106 >> e7ef94cb6fc >> Deleted: sha256:67e4c09745aa1fd979f24bfae0812811bd44377dafd43130c0c27 >> 69e820d252a >> Deleted: sha256:958624053829cf5b33921f84ad226d1835a271befa6856f591fc0 >> b4e1981c349 >> Deleted: sha256:a59579b1b2fd24a0570169c6a97d2ba86c88f44a9ac159c792f37 >> 71c9d3585a7 >> Deleted: sha256:4edbd39dc9e6dcb72d14b8e207b6705bace589ce1dc8cfe4525ff >> ac3d216f471 >> Deleted: sha256:92fc49f6ea70503f089441fdcdfdcf8ebc3875060deefaf5f355c >> 4a9b45314e8 >> Build step 'Execute shell' marked build as failure >> > >