See <https://builds.apache.org/job/Mesos-Buildbot/BUILDTOOL=cmake,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A14.04,label_exp=(ubuntu)&&(!ubuntu-us1)&&(!ubuntu-eu2)&&(!qnode3)&&(!H23)/4492/display/redirect?page=changes>
Changes: [bmahler] Optimized resources logging in master. ------------------------------------------ [...truncated 21.18 MB...] 3: I1122 21:47:57.279350 18357 slave.cpp:1007] New master detected at [email protected]:40411 3: I1122 21:47:57.279460 18357 slave.cpp:1042] Detecting new master 3: I1122 21:47:57.291805 18361 slave.cpp:1069] Authenticating with master [email protected]:40411 3: I1122 21:47:57.291891 18361 slave.cpp:1078] Using default CRAM-MD5 authenticatee 3: I1122 21:47:57.292191 18368 authenticatee.cpp:121] Creating new client SASL connection 3: I1122 21:47:57.292601 18367 master.cpp:8312] Authenticating slave(546)@172.17.0.4:40411 3: I1122 21:47:57.292743 18370 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(1097)@172.17.0.4:40411 3: I1122 21:47:57.293062 18359 authenticator.cpp:98] Creating new server SASL connection 3: I1122 21:47:57.293306 18372 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 3: I1122 21:47:57.293336 18372 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' 3: I1122 21:47:57.293462 18375 authenticator.cpp:204] Received SASL authentication start 3: I1122 21:47:57.293534 18375 authenticator.cpp:326] Authentication requires more steps 3: I1122 21:47:57.293644 18356 authenticatee.cpp:259] Received SASL authentication step 3: I1122 21:47:57.293781 18360 authenticator.cpp:232] Received SASL authentication step 3: I1122 21:47:57.293817 18360 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'dd25ef0f766f' server FQDN: 'dd25ef0f766f' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 3: I1122 21:47:57.293831 18360 auxprop.cpp:181] Looking up auxiliary property '*userPassword' 3: I1122 21:47:57.293887 18360 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' 3: I1122 21:47:57.293907 18360 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'dd25ef0f766f' server FQDN: 'dd25ef0f766f' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 3: I1122 21:47:57.293921 18360 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true 3: I1122 21:47:57.293928 18360 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true 3: I1122 21:47:57.293949 18360 authenticator.cpp:318] Authentication success 3: I1122 21:47:57.294040 18355 authenticatee.cpp:299] Authentication success 3: I1122 21:47:57.294121 18369 master.cpp:8342] Successfully authenticated principal 'test-principal' at slave(546)@172.17.0.4:40411 3: I1122 21:47:57.294183 18366 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(1097)@172.17.0.4:40411 3: I1122 21:47:57.294364 18355 slave.cpp:1161] Successfully authenticated with master [email protected]:40411 3: I1122 21:47:57.294620 18355 slave.cpp:1685] Will retry registration in 9.298672ms if necessary 3: I1122 21:47:57.294865 18373 master.cpp:6036] Received register agent message from slave(546)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:57.294904 18373 master.cpp:3872] Authorizing agent with principal 'test-principal' 3: I1122 21:47:57.295337 18354 master.cpp:6098] Authorized registration of agent at slave(546)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:57.295445 18354 master.cpp:6191] Registering agent at slave(546)@172.17.0.4:40411 (dd25ef0f766f) with id a8075b7a-0017-4079-ae49-b94e62ac0556-S0 3: I1122 21:47:57.295799 18371 registrar.cpp:495] Applied 1 operations in 79852ns; attempting to update the registry 3: I1122 21:47:57.296430 18371 registrar.cpp:552] Successfully updated the registry in 572928ns 3: I1122 21:47:57.296679 18368 master.cpp:6240] Admitted agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:57.297307 18368 master.cpp:6276] Registered agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) with cpus:2; mem:1024; disk:1024; ports:[31000-32000] 3: I1122 21:47:57.297392 18370 slave.cpp:1207] Registered with master [email protected]:40411; given agent ID a8075b7a-0017-4079-ae49-b94e62ac0556-S0 3: I1122 21:47:57.297487 18375 task_status_update_manager.cpp:188] Resuming sending task status updates 3: I1122 21:47:57.297641 18359 hierarchical.cpp:600] Added agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 (dd25ef0f766f) with cpus:2; mem:1024; disk:1024; ports:[31000-32000] (allocated: {}) 3: I1122 21:47:57.297700 18370 slave.cpp:1227] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/slave.info' 3: I1122 21:47:57.298102 18370 slave.cpp:1298] Forwarding total oversubscribed resources {} 3: I1122 21:47:57.298322 18365 master.cpp:7091] Received update of agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) with total oversubscribed resources {} 3: I1122 21:47:57.298511 18365 master.cpp:7109] Ignoring update on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) as it reports no changes 3: I1122 21:47:57.298717 18359 hierarchical.cpp:1457] Performed allocation for 1 agents in 945344ns 3: I1122 21:47:57.299125 18360 master.cpp:8142] Sending 1 offers to framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (default) at [email protected]:40411 3: I1122 21:47:57.299787 18369 sched.cpp:921] Scheduler::resourceOffers took 161734ns 3: I1122 21:47:57.301589 18363 master.cpp:10057] Removing offer a8075b7a-0017-4079-ae49-b94e62ac0556-O0 3: I1122 21:47:57.301731 18363 master.cpp:4230] Processing ACCEPT call for offers: [ a8075b7a-0017-4079-ae49-b94e62ac0556-O0 ] on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) for framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (default) at [email protected]:40411 3: I1122 21:47:57.301861 18363 master.cpp:3599] Authorizing framework principal 'test-principal' to launch task 9518193c-1ca9-400e-ba80-063e5376a860 3: I1122 21:47:57.303768 18352 master.cpp:10808] Adding task 9518193c-1ca9-400e-ba80-063e5376a860 with resources cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:57.304147 18352 master.cpp:4978] Launching task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (default) at [email protected]:40411 with resources [{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:57.305075 18371 slave.cpp:1806] Got assigned task '9518193c-1ca9-400e-ba80-063e5376a860' for framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.305265 18371 slave.cpp:7663] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/framework.info' 3: I1122 21:47:57.305629 18371 slave.cpp:7674] Checkpointing framework pid '[email protected]:40411' to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/framework.pid' 3: I1122 21:47:57.306293 18371 slave.cpp:2074] Authorizing task '9518193c-1ca9-400e-ba80-063e5376a860' for framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.306347 18371 slave.cpp:7270] Authorizing framework principal 'test-principal' to launch task 9518193c-1ca9-400e-ba80-063e5376a860 3: I1122 21:47:57.307178 18375 slave.cpp:2242] Launching task '9518193c-1ca9-400e-ba80-063e5376a860' for framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.307834 18375 paths.cpp:621] Trying to chown '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539' to user 'mesos' 3: I1122 21:47:57.308105 18375 slave.cpp:8368] Checkpointing ExecutorInfo to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/executor.info' 3: I1122 21:47:57.308770 18375 slave.cpp:7744] Launching executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 with resources [{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"mem","scalar":{"value":32.0},"type":"SCALAR"}] in work directory '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539' 3: I1122 21:47:57.309739 18375 slave.cpp:8411] Checkpointing TaskInfo to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539/tasks/9518193c-1ca9-400e-ba80-063e5376a860/task.info' 3: I1122 21:47:57.310204 18375 slave.cpp:2470] Queued task '9518193c-1ca9-400e-ba80-063e5376a860' for executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.310293 18375 slave.cpp:958] Successfully attached '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539' to virtual path '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/latest' 3: I1122 21:47:57.310323 18375 slave.cpp:958] Successfully attached '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539' to virtual path '/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/latest' 3: I1122 21:47:57.310348 18375 slave.cpp:958] Successfully attached '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539' to virtual path '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539' 3: I1122 21:47:57.310731 18375 slave.cpp:2933] Launching container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 for executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.310961 18365 containerizer.cpp:1195] Starting container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 3: I1122 21:47:57.311759 18365 containerizer.cpp:1367] Checkpointed ContainerConfig at '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/containers/77f2c2c4-49ca-4aa0-b950-28d9ecf37539/config' 3: I1122 21:47:57.311781 18365 containerizer.cpp:2919] Transitioning the state of container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 from PROVISIONING to PREPARING 3: I1122 21:47:57.315428 18373 containerizer.cpp:1836] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"command":{"arguments":["mesos-executor","--launcher_dir=\/mesos\/build\/src"],"shell":false,"value":"\/mesos\/build\/src\/mesos-executor"},"environment":{"variables":[{"name":"LIBPROCESS_PORT","type":"VALUE","value":"0"},{"name":"MESOS_AGENT_ENDPOINT","type":"VALUE","value":"172.17.0.4:40411"},{"name":"MESOS_CHECKPOINT","type":"VALUE","value":"1"},{"name":"MESOS_DIRECTORY","type":"VALUE","value":"\/tmp\/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36\/slaves\/a8075b7a-0017-4079-ae49-b94e62ac0556-S0\/frameworks\/a8075b7a-0017-4079-ae49-b94e62ac0556-0000\/executors\/9518193c-1ca9-400e-ba80-063e5376a860\/runs\/77f2c2c4-49ca-4aa0-b950-28d9ecf37539"},{"name":"MESOS_EXECUTOR_AUTHENTICATION_TOKEN","type":"VALUE","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjaWQiOiI3N2YyYzJjNC00OWNhLTRhYTAtYjk1MC0yOGQ5ZWNmMzc1MzkiLCJlaWQiOiI5NTE4MTkzYy0xY2E5LTQwMGUtYmE4MC0wNjNlNTM3NmE4NjAiLCJmaWQiOiJhODA3NWI3YS0wMDE3LTQwNzktYWU0OS1iOTRlNjJhYzA1NTYtMDAwMCJ9.Q9dgLdHd18MtX81eLIZBufHQI_Jo5mki3ldAiAF3Eyo"},{"name":"MESOS_EXECUTOR_ID","type":"VALUE","value":"9518193c-1ca9-400e-ba80-063e5376a860"},{"name":"MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD","type":"VALUE","value":"5secs"},{"name":"MESOS_FRAMEWORK_ID","type":"VALUE","value":"a8075b7a-0017-4079-ae49-b94e62ac0556-0000"},{"name":"MESOS_HTTP_COMMAND_EXECUTOR","type":"VALUE","value":"0"},{"name":"MESOS_RECOVERY_TIMEOUT","type":"VALUE","value":"15mins"},{"name":"MESOS_SLAVE_ID","type":"VALUE","value":"a8075b7a-0017-4079-ae49-b94e62ac0556-S0"},{"name":"MESOS_SLAVE_PID","type":"VALUE","value":"slave(546)@172.17.0.4:40411"},{"name":"MESOS_SUBSCRIPTION_BACKOFF_MAX","type":"VALUE","value":"2secs"},{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/tmp\/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36\/slaves\/a8075b7a-0017-4079-ae49-b94e62ac0556-S0\/frameworks\/a8075b7a-0017-4079-ae49-b94e62ac0556-0000\/executors\/9518193c-1ca9-400e-ba80-063e5376a860\/runs\/77f2c2c4-49ca-4aa0-b950-28d9ecf37539"}]},"task_environment":{},"user":"mesos","working_directory":"\/tmp\/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36\/slaves\/a8075b7a-0017-4079-ae49-b94e62ac0556-S0\/frameworks\/a8075b7a-0017-4079-ae49-b94e62ac0556-0000\/executors\/9518193c-1ca9-400e-ba80-063e5376a860\/runs\/77f2c2c4-49ca-4aa0-b950-28d9ecf37539"}" --pipe_read="8" --pipe_write="9" --runtime_directory="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/containers/77f2c2c4-49ca-4aa0-b950-28d9ecf37539" --unshare_namespace_mnt="false"' 3: I1122 21:47:57.317596 18373 launcher.cpp:140] Forked child with pid '23981' for container '77f2c2c4-49ca-4aa0-b950-28d9ecf37539' 3: I1122 21:47:57.317759 18373 containerizer.cpp:1928] Checkpointing container's forked pid 23981 to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539/pids/forked.pid' 3: I1122 21:47:57.318398 18373 containerizer.cpp:2919] Transitioning the state of container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 from PREPARING to ISOLATING 3: I1122 21:47:57.319747 18373 containerizer.cpp:2919] Transitioning the state of container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 from ISOLATING to FETCHING 3: I1122 21:47:57.319890 18356 fetcher.cpp:379] Starting to fetch URIs for container: 77f2c2c4-49ca-4aa0-b950-28d9ecf37539, directory: /tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539 3: I1122 21:47:57.320798 18353 containerizer.cpp:2919] Transitioning the state of container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 from FETCHING to RUNNING 3: I1122 21:47:57.528498 23982 exec.cpp:162] Version: 1.5.0 3: I1122 21:47:57.537600 18354 slave.cpp:4066] Got registration for executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 from executor(1)@172.17.0.4:43915 3: I1122 21:47:57.538239 18354 slave.cpp:4152] Checkpointing executor pid 'executor(1)@172.17.0.4:43915' to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539/pids/libprocess.pid' 3: I1122 21:47:57.540500 18356 slave.cpp:2672] Sending queued task '9518193c-1ca9-400e-ba80-063e5376a860' to executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 at executor(1)@172.17.0.4:43915 3: I1122 21:47:57.543064 24002 exec.cpp:237] Executor registered on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 3: I1122 21:47:57.546569 23984 executor.cpp:171] Received SUBSCRIBED event 3: I1122 21:47:57.547793 23984 executor.cpp:175] Subscribed executor on dd25ef0f766f 3: I1122 21:47:57.548002 23984 executor.cpp:171] Received LAUNCH event 3: I1122 21:47:57.549608 23984 executor.cpp:637] Starting task 9518193c-1ca9-400e-ba80-063e5376a860 3: I1122 21:47:57.552371 18362 slave.cpp:4520] Handling status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 from executor(1)@172.17.0.4:43915 3: I1122 21:47:57.554020 18354 task_status_update_manager.cpp:328] Received task status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.554090 18354 task_status_update_manager.cpp:507] Creating StatusUpdate stream for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.554975 18354 task_status_update_manager.cpp:842] Checkpointing UPDATE for task status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.555275 18354 task_status_update_manager.cpp:383] Forwarding task status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to the agent 3: I1122 21:47:57.555500 18375 slave.cpp:5003] Forwarding the update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to [email protected]:40411 3: I1122 21:47:57.556005 18375 slave.cpp:4896] Task status update manager successfully handled status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.556063 18375 slave.cpp:4912] Sending acknowledgement for status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to executor(1)@172.17.0.4:43915 3: I1122 21:47:57.556422 18375 slave.cpp:883] Agent terminating 3: I1122 21:47:57.556741 18365 master.cpp:1311] Agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) disconnected 3: I1122 21:47:57.556766 18365 master.cpp:3370] Disconnecting agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:57.556787 18351 containerizer.cpp:301] Using isolation { environment_secret, posix/cpu, posix/mem, filesystem/posix, network/cni } 3: I1122 21:47:57.556838 18365 master.cpp:3389] Deactivating agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:57.556929 18353 hierarchical.cpp:697] Agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 deactivated 3: W1122 21:47:57.557349 18351 backend.cpp:76] Failed to create 'aufs' backend: AufsBackend requires root privileges 3: W1122 21:47:57.557488 18351 backend.cpp:76] Failed to create 'bind' backend: BindBackend requires root privileges 3: I1122 21:47:57.557528 18351 provisioner.cpp:259] Using default backend 'copy' 3: I1122 21:47:57.562405 23984 executor.cpp:477] Running '/mesos/build/src/mesos-containerizer launch <POSSIBLY-SENSITIVE-DATA>' 3: W1122 21:47:57.563962 18351 process.cpp:2756] Attempted to spawn already running process [email protected]:40411 3: I1122 21:47:57.564280 18351 cluster.cpp:448] Creating default 'local' authorizer 3: I1122 21:47:57.565461 23984 executor.cpp:650] Forked command at 24008 3: I1122 21:47:57.566445 18360 slave.cpp:262] Mesos agent started on (547)@172.17.0.4:40411 3: I1122 21:47:57.566483 18360 slave.cpp:263] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://"; --appc_store_dir="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/credential" --default_role="*" --disallow_sharing_agent_pid_namespace="false" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_reregistration_timeout="2secs" --executor_secret_key="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/executor_secret_key" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36" --zk_session_timeout="10secs" 3: I1122 21:47:57.567101 18360 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/credential' 3: I1122 21:47:57.567338 18360 slave.cpp:295] Agent using credential for: test-principal 3: I1122 21:47:57.567368 18360 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_s3xvX2/http_credentials' 3: I1122 21:47:57.567742 18360 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor' 3: I1122 21:47:57.571337 18360 http.cpp:1066] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor' 3: I1122 21:47:57.571599 18360 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly' 3: I1122 21:47:57.571686 18376 process.cpp:2578] Failed to deliver libprocess message to /slave(546)/mesos.internal.StatusUpdateMessage 3: I1122 21:47:57.571710 18360 http.cpp:1066] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly' 3: I1122 21:47:57.571900 18360 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite' 3: I1122 21:47:57.572008 18360 http.cpp:1066] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite' 3: I1122 21:47:57.573596 18360 slave.cpp:593] Agent resources: [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] 3: I1122 21:47:57.573839 18360 slave.cpp:601] Agent attributes: [ ] 3: I1122 21:47:57.573849 18360 slave.cpp:610] Agent hostname: dd25ef0f766f 3: I1122 21:47:57.574067 18365 task_status_update_manager.cpp:181] Pausing sending task status updates 3: I1122 21:47:57.575981 18358 state.cpp:64] Recovering state from '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta' 3: I1122 21:47:57.576079 18358 state.cpp:722] No committed checkpointed resources found at '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/resources/resources.info' 3: I1122 21:47:57.580032 18353 slave.cpp:6546] Recovering framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.580164 18353 slave.cpp:7858] Recovering executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.580925 18359 task_status_update_manager.cpp:207] Recovering task status update manager 3: I1122 21:47:57.580950 18359 task_status_update_manager.cpp:215] Recovering executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.581012 18359 task_status_update_manager.cpp:507] Creating StatusUpdate stream for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.581590 18359 task_status_update_manager.cpp:818] Replaying task status update stream for task 9518193c-1ca9-400e-ba80-063e5376a860 3: I1122 21:47:57.582073 18368 containerizer.cpp:668] Recovering containerizer 3: I1122 21:47:57.582216 18368 containerizer.cpp:725] Recovering container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 for executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.585784 18352 provisioner.cpp:455] Provisioner recovery complete 3: I1122 21:47:57.586766 18353 slave.cpp:6333] Sending reconnect request to executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 at executor(1)@172.17.0.4:43915 3: I1122 21:47:57.588212 23991 exec.cpp:283] Received reconnect request from agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 3: I1122 21:47:57.590308 18373 slave.cpp:4230] Received re-registration message from executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.590689 18373 slave.cpp:4520] Handling status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 from executor(1)@172.17.0.4:43915 3: I1122 21:47:57.592108 23995 exec.cpp:260] Executor re-registered on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 3: I1122 21:47:57.592346 18370 task_status_update_manager.cpp:328] Received task status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.592437 18370 task_status_update_manager.cpp:842] Checkpointing UPDATE for task status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.592820 18354 slave.cpp:4896] Task status update manager successfully handled status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:57.592885 18354 slave.cpp:4912] Sending acknowledgement for status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to executor(1)@172.17.0.4:43915 3: I1122 21:47:57.594588 23992 executor.cpp:171] Received SUBSCRIBED event 3: I1122 21:47:57.594614 23992 executor.cpp:175] Subscribed executor on dd25ef0f766f 3: I1122 21:47:58.258062 18356 hierarchical.cpp:1457] Performed allocation for 1 agents in 239807ns 3: I1122 21:47:59.261754 18372 hierarchical.cpp:1457] Performed allocation for 1 agents in 241047ns 3: I1122 21:47:59.588990 18354 slave.cpp:4448] Cleaning up un-reregistered executors 3: I1122 21:47:59.589139 18354 slave.cpp:6449] Finished recovery 3: I1122 21:47:59.590411 18368 task_status_update_manager.cpp:181] Pausing sending task status updates 3: I1122 21:47:59.590411 18362 slave.cpp:1007] New master detected at [email protected]:40411 3: I1122 21:47:59.590495 18362 slave.cpp:1042] Detecting new master 3: I1122 21:47:59.598701 18369 slave.cpp:1069] Authenticating with master [email protected]:40411 3: I1122 21:47:59.598814 18369 slave.cpp:1078] Using default CRAM-MD5 authenticatee 3: I1122 21:47:59.599187 18373 authenticatee.cpp:121] Creating new client SASL connection 3: I1122 21:47:59.599544 18357 master.cpp:8312] Authenticating slave(547)@172.17.0.4:40411 3: I1122 21:47:59.599663 18356 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(1098)@172.17.0.4:40411 3: I1122 21:47:59.600020 18352 authenticator.cpp:98] Creating new server SASL connection 3: I1122 21:47:59.600260 18355 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 3: I1122 21:47:59.600283 18355 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' 3: I1122 21:47:59.600401 18360 authenticator.cpp:204] Received SASL authentication start 3: I1122 21:47:59.600464 18360 authenticator.cpp:326] Authentication requires more steps 3: I1122 21:47:59.600594 18364 authenticatee.cpp:259] Received SASL authentication step 3: I1122 21:47:59.600731 18358 authenticator.cpp:232] Received SASL authentication step 3: I1122 21:47:59.600759 18358 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'dd25ef0f766f' server FQDN: 'dd25ef0f766f' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 3: I1122 21:47:59.600776 18358 auxprop.cpp:181] Looking up auxiliary property '*userPassword' 3: I1122 21:47:59.600821 18358 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' 3: I1122 21:47:59.600847 18358 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'dd25ef0f766f' server FQDN: 'dd25ef0f766f' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 3: I1122 21:47:59.600862 18358 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true 3: I1122 21:47:59.600869 18358 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true 3: I1122 21:47:59.600888 18358 authenticator.cpp:318] Authentication success 3: I1122 21:47:59.600996 18366 authenticatee.cpp:299] Authentication success 3: I1122 21:47:59.601071 18365 master.cpp:8342] Successfully authenticated principal 'test-principal' at slave(547)@172.17.0.4:40411 3: I1122 21:47:59.601088 18363 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(1098)@172.17.0.4:40411 3: I1122 21:47:59.601287 18371 slave.cpp:1161] Successfully authenticated with master [email protected]:40411 3: I1122 21:47:59.601872 18371 slave.cpp:1685] Will retry registration in 17.623879ms if necessary 3: I1122 21:47:59.602418 18370 master.cpp:6365] Received re-register agent message from agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:59.602455 18370 master.cpp:3872] Authorizing agent with principal 'test-principal' 3: I1122 21:47:59.603037 18370 master.cpp:6436] Authorized re-registration of agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:59.603132 18370 master.cpp:6501] Re-registering agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(546)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:59.603569 18367 slave.cpp:1343] Re-registered with master [email protected]:40411 3: I1122 21:47:59.603579 18362 hierarchical.cpp:685] Agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 reactivated 3: I1122 21:47:59.603636 18370 master.cpp:6608] Sending updated checkpointed resources {} to agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:59.603668 18375 task_status_update_manager.cpp:188] Resuming sending task status updates 3: I1122 21:47:59.603703 18367 slave.cpp:1400] Forwarding total oversubscribed resources {} 3: W1122 21:47:59.603708 18375 task_status_update_manager.cpp:195] Resending task status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.603761 18375 task_status_update_manager.cpp:383] Forwarding task status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to the agent 3: I1122 21:47:59.603932 18369 master.cpp:7091] Received update of agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) with total oversubscribed resources {} 3: I1122 21:47:59.603962 18367 slave.cpp:3441] Updating info for framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 with pid updated to [email protected]:40411 3: I1122 21:47:59.604060 18367 slave.cpp:7663] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/framework.info' 3: I1122 21:47:59.604212 18369 master.cpp:7109] Ignoring update on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) as it reports no changes 3: I1122 21:47:59.604585 18367 slave.cpp:7674] Checkpointing framework pid '[email protected]:40411' to '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/meta/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/framework.pid' 3: I1122 21:47:59.604950 18357 task_status_update_manager.cpp:188] Resuming sending task status updates 3: W1122 21:47:59.604975 18357 task_status_update_manager.cpp:195] Resending task status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.605006 18357 task_status_update_manager.cpp:383] Forwarding task status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to the agent 3: I1122 21:47:59.605020 18367 slave.cpp:3517] Ignoring new checkpointed resources identical to the current version: {} 3: I1122 21:47:59.605167 18367 slave.cpp:5003] Forwarding the update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to [email protected]:40411 3: I1122 21:47:59.605438 18367 slave.cpp:5003] Forwarding the update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to [email protected]:40411 3: I1122 21:47:59.605520 18373 master.cpp:7274] Status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 from agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:59.605564 18373 master.cpp:7336] Forwarding status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.605754 18373 master.cpp:9529] Updating the state of task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (latest state: TASK_RUNNING, status update state: TASK_STARTING) 3: I1122 21:47:59.606010 18373 master.cpp:7274] Status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 from agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:59.606045 18374 sched.cpp:1029] Scheduler::statusUpdate took 131513ns 3: I1122 21:47:59.606061 18373 master.cpp:7336] Forwarding status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.606242 18373 master.cpp:9529] Updating the state of task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (latest state: TASK_RUNNING, status update state: TASK_STARTING) 3: I1122 21:47:59.606438 18374 sched.cpp:1029] Scheduler::statusUpdate took 23773ns 3: I1122 21:47:59.606468 18373 master.cpp:5793] Processing ACKNOWLEDGE call 11c8e8b6-daf6-4526-acd1-38b102cba491 for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (default) at [email protected]:40411 on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 3: I1122 21:47:59.606489 18351 sched.cpp:2009] Asked to stop the driver 3: I1122 21:47:59.606580 18374 sched.cpp:1191] Stopping framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.606791 18359 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.606914 18373 master.cpp:5793] Processing ACKNOWLEDGE call 11c8e8b6-daf6-4526-acd1-38b102cba491 for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (default) at [email protected]:40411 on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 3: I1122 21:47:59.606914 18359 task_status_update_manager.cpp:842] Checkpointing ACK for task status update TASK_STARTING (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.607095 18373 master.cpp:8819] Processing TEARDOWN call for framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (default) at [email protected]:40411 3: I1122 21:47:59.607084 18359 task_status_update_manager.cpp:383] Forwarding task status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to the agent 3: I1122 21:47:59.607115 18373 master.cpp:8831] Removing framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (default) at [email protected]:40411 3: I1122 21:47:59.607125 18373 master.cpp:3333] Deactivating framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (default) at [email protected]:40411 3: I1122 21:47:59.607266 18363 hierarchical.cpp:419] Deactivated framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.607296 18373 master.cpp:9529] Updating the state of task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED) 3: I1122 21:47:59.607317 18372 slave.cpp:5003] Forwarding the update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 to [email protected]:40411 3: I1122 21:47:59.607388 18359 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: W1122 21:47:59.607494 18359 task_status_update_manager.cpp:771] Duplicate task status update acknowledgment (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.607554 18372 slave.cpp:3270] Asked to shut down framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 by [email protected]:40411 3: I1122 21:47:59.607584 18372 slave.cpp:3295] Shutting down framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.607648 18372 slave.cpp:5879] Shutting down executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 at executor(1)@172.17.0.4:43915 3: I1122 21:47:59.607909 18373 master.cpp:9629] Removing task 9518193c-1ca9-400e-ba80-063e5376a860 with resources cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:59.608108 18372 slave.cpp:3804] Task status update manager successfully handled status update acknowledgement (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: E1122 21:47:59.608191 18372 slave.cpp:3797] Failed to handle status update acknowledgement (UUID: 11c8e8b6-daf6-4526-acd1-38b102cba491) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000: Duplicate task status acknowledgement 3: I1122 21:47:59.608781 18373 master.cpp:7274] Status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 from agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) 3: W1122 21:47:59.608826 18373 master.cpp:7289] Received status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 from agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) for an unknown framework 3: W1122 21:47:59.608866 18373 master.cpp:7304] Could not lookup task for status update TASK_RUNNING (UUID: 5c3a718f-ca86-4908-9193-2c3ecd565229) for task 9518193c-1ca9-400e-ba80-063e5376a860 of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 from agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 at slave(547)@172.17.0.4:40411 (dd25ef0f766f) 3: I1122 21:47:59.608943 18371 hierarchical.cpp:1132] Recovered cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] (total: cpus:2; mem:1024; disk:1024; ports:[31000-32000], allocated: {}) on agent a8075b7a-0017-4079-ae49-b94e62ac0556-S0 from framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.609052 23996 exec.cpp:435] Executor asked to shutdown 3: I1122 21:47:59.609093 18363 containerizer.cpp:2322] Destroying container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 in RUNNING state 3: I1122 21:47:59.609117 18363 containerizer.cpp:2919] Transitioning the state of container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 from RUNNING to DESTROYING 3: I1122 21:47:59.609239 18371 hierarchical.cpp:358] Removed framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 3: I1122 21:47:59.609387 18363 launcher.cpp:156] Asked to destroy container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 3: I1122 21:47:59.609516 23996 executor.cpp:171] Received SHUTDOWN event 3: I1122 21:47:59.609551 23996 executor.cpp:747] Shutting down 3: I1122 21:47:59.609593 23996 executor.cpp:854] Sending SIGTERM to process tree at pid 24008 3: I1122 21:47:59.621660 18358 slave.cpp:5135] Got exited event for executor(1)@172.17.0.4:43915 3: I1122 21:47:59.655308 18371 containerizer.cpp:2773] Container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 has exited 3: I1122 21:47:59.657469 18358 provisioner.cpp:557] Ignoring destroy request for unknown container 77f2c2c4-49ca-4aa0-b950-28d9ecf37539 3: I1122 21:47:59.658238 18353 slave.cpp:5544] Executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 terminated with signal Killed 3: I1122 21:47:59.658304 18353 slave.cpp:5648] Cleaning up executor '9518193c-1ca9-400e-ba80-063e5376a860' of framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 at executor(1)@172.17.0.4:43915 3: I1122 21:47:59.658772 18372 gc.cpp:90] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860/runs/77f2c2c4-49ca-4aa0-b950-28d9ecf37539' for gc 6.9999923764days in the future 3: I1122 21:47:59.659006 18372 gc.cpp:90] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverTaskStatusUpdateManager_0rnh36/slaves/a8075b7a-0017-4079-ae49-b94e62ac0556-S0/frameworks/a8075b7a-0017-4079-ae49-b94e62ac0556-0000/executors/9518193c-1ca9-400e-ba80-063e5376a860' for gc 6.99999237417778days in the future 3: I1122 21:47:59.659088 18353 slave.cpp:5755] Cleaning up framework a8075b7a-0017-4079-ae49-b94e62ac0556-0000 write /dev/stdout: resource temporarily unavailable + docker rmi mesos-1511385529-3970 Error response from daemon: conflict: unable to remove repository reference "mesos-1511385529-3970" (must force) - container dd25ef0f766f is using its referenced image 3c2287a8b04f Process leaked file descriptors. See https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for more information Build step 'Execute shell' marked build as failure
