[
https://issues.apache.org/jira/browse/MESOS-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Xu updated MESOS-735:
-------------------------
Description:
sudo MESOS_VERBOSE=1 GLOG_v=1 ./bin/mesos-tests.sh
--gtest_filter="SlaveRecoveryTest/0.RegisterDisconnectedSlave"
--gtest_repeat=1000 --gest_break_on_failure
Error:
../../src/tests/slave_recovery_tests.cpp:1647: Failure
Failed to wait 10secs for status
The TASK_KILLED status for task "3775fcd5-4776-4e46-aacb-93db531757b1" is from
a previous iteration of the test.
Log:
[ RUN ] SlaveRecoveryTest/0.RegisterDisconnectedSlave
I1014 17:13:07.347805 61261 master.cpp:284] Master started on 10.35.12.124:58876
I1014 17:13:07.347967 61261 master.cpp:299] Master ID:
201310141713-2081170186-58876-61234
I1014 17:13:07.348089 61261 master.cpp:302] Master only allowing authenticated
frameworks to register!
I1014 17:13:07.348464 61259 hierarchical_allocator_process.hpp:302]
Initializing hierarchical allocator process with master :
master@10.35.12.124:58876
I1014 17:13:07.348469 61251 master.cpp:85] No whitelist given. Advertising
offers for all slaves
I1014 17:13:07.348611 61255 slave.cpp:108] Slave started on
168)@10.35.12.124:58876
I1014 17:13:07.348711 61261 master.cpp:697] Elected as master!
I1014 17:13:07.348975 61255 slave.cpp:208] Slave resources: cpus(*):2;
mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I1014 17:13:07.349717 61262 process_isolator.cpp:314] Recovering isolator
I1014 17:13:07.349750 61255 slave.cpp:547] New master detected at
master@10.35.12.124:58876
I1014 17:13:07.350041 61255 slave.cpp:562] Postponing registration until
recovery is complete
I1014 17:13:07.350059 61260 status_update_manager.cpp:157] New master detected
at master@10.35.12.124:58876
I1014 17:13:07.350128 61255 slave.cpp:399] Finished recovery
I1014 17:13:07.350483 61256 master.cpp:1248] Attempting to register slave on
smfd-aki-27-sr1.devel.twitter.com at slave(168)@10.35.12.124:58876
I1014 17:13:07.350579 61256 master.cpp:2502] Adding slave
201310141713-2081170186-58876-61234-0 at smfd-aki-27-sr1.devel.twitter.com with
cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I1014 17:13:07.350703 61258 slave.cpp:613] Registered with master
master@10.35.12.124:58876; given slave ID 201310141713-2081170186-58876-61234-0
I1014 17:13:07.350824 61250 hierarchical_allocator_process.hpp:445] Added slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com) with
cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2;
mem(*):1024; disk(*):1024; ports(*):[31000-32000] available)
I1014 17:13:07.350941 61258 slave.cpp:626] Checkpointing SlaveInfo to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/slave.info'
I1014 17:13:07.350998 61250 hierarchical_allocator_process.hpp:708] Performed
allocation for slave 201310141713-2081170186-58876-61234-0 in 7.03us
I1014 17:13:07.351054 61254 sched.cpp:195] New master at
master@10.35.12.124:58876
I1014 17:13:07.351310 61254 sched.cpp:281] Authenticating with master
master@10.35.12.124:58876
I1014 17:13:07.351511 61263 authenticatee.hpp:124] Creating new client SASL
connection
I1014 17:13:07.351646 61253 master.cpp:1723] Authenticating framework at
scheduler(168)@10.35.12.124:58876
I1014 17:13:07.351886 61257 authenticator.hpp:140] Creating new server SASL
connection
I1014 17:13:07.352033 61253 authenticatee.hpp:212] Received SASL authentication
mechanisms: CRAM-MD5
I1014 17:13:07.352105 61253 authenticatee.hpp:238] Attempting to authenticate
with mechanism 'CRAM-MD5'
I1014 17:13:07.352202 61255 authenticator.hpp:243] Received SASL authentication
start
I1014 17:13:07.352306 61255 authenticator.hpp:325] Authentication requires more
steps
I1014 17:13:07.352394 61255 authenticatee.hpp:258] Received SASL authentication
step
I1014 17:13:07.352504 61255 authenticator.hpp:271] Received SASL authentication
step
I1014 17:13:07.352581 61255 auxprop.cpp:81] Request to lookup properties for
user: 'test-principal' realm: 'smfd-aki-27-sr1.devel.twitter.com' server FQDN:
'smfd-aki-27-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false
SASL_AUXPROP_AUTHZID: false
I1014 17:13:07.352654 61255 auxprop.cpp:153] Looking up auxiliary property
'*userPassword'
I1014 17:13:07.352728 61255 auxprop.cpp:153] Looking up auxiliary property
'*cmusaslsecretCRAM-MD5'
I1014 17:13:07.352825 61255 auxprop.cpp:81] Request to lookup properties for
user: 'test-principal' realm: 'smfd-aki-27-sr1.devel.twitter.com' server FQDN:
'smfd-aki-27-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false
SASL_AUXPROP_AUTHZID: true
I1014 17:13:07.352893 61255 auxprop.cpp:103] Skipping auxiliary property
'*userPassword' since SASL_AUXPROP_AUTHZID == true
I1014 17:13:07.352957 61255 auxprop.cpp:103] Skipping auxiliary property
'*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1014 17:13:07.353035 61255 authenticator.hpp:317] Authentication success
I1014 17:13:07.353150 61253 authenticatee.hpp:298] Authentication success
I1014 17:13:07.353183 61254 master.cpp:1763] Successfully authenticated
framework at scheduler(168)@10.35.12.124:58876
I1014 17:13:07.353281 61253 sched.cpp:326] Successfully authenticated with
master master@10.35.12.124:58876
I1014 17:13:07.353430 61254 master.cpp:768] Received registration request from
scheduler(168)@10.35.12.124:58876
I1014 17:13:07.353574 61254 master.cpp:786] Registering framework
201310141713-2081170186-58876-61234-0000 at scheduler(168)@10.35.12.124:58876
I1014 17:13:07.353706 61259 sched.cpp:365] Framework registered with
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.353735 61255 hierarchical_allocator_process.hpp:332] Added
framework 201310141713-2081170186-58876-61234-0000
I1014 17:13:07.353799 61259 sched.cpp:379] Scheduler::registered took 9.296us
I1014 17:13:07.353881 61255 hierarchical_allocator_process.hpp:752] Offering
cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 to framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.354061 61255 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 212.58us
I1014 17:13:07.354136 61262 master.hpp:389] Adding offer
201310141713-2081170186-58876-61234-0 with resources cpus(*):2; mem(*):1024;
disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
I1014 17:13:07.354249 61262 master.cpp:1689] Sending 1 offers to framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.354419 61260 sched.cpp:472] Scheduler::resourceOffers took
22.071us
I1014 17:13:07.355800 61250 master.cpp:2015] Processing reply for offer
201310141713-2081170186-58876-61234-0 on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com) for
framework 201310141713-2081170186-58876-61234-0000
I1014 17:13:07.355950 61250 master.hpp:361] Adding task
02786c98-93b9-4a26-ae6d-deefd2708b94 with resources cpus(*):2; mem(*):1024;
disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
I1014 17:13:07.356057 61250 master.cpp:2139] Launching task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 with resources cpus(*):2; mem(*):1024;
disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
I1014 17:13:07.356216 61257 slave.cpp:786] Got assigned task
02786c98-93b9-4a26-ae6d-deefd2708b94 for framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.356230 61250 master.hpp:399] Removing offer
201310141713-2081170186-58876-61234-0 with resources cpus(*):2; mem(*):1024;
disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
I1014 17:13:07.356366 61257 slave.cpp:2844] Checkpointing FrameworkInfo to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/framework.info'
I1014 17:13:07.356609 61257 slave.cpp:2851] Checkpointing framework pid
'scheduler(168)@10.35.12.124:58876' to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/framework.pid'
I1014 17:13:07.356909 61257 slave.cpp:897] Launching task
02786c98-93b9-4a26-ae6d-deefd2708b94 for framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.358155 61257 slave.cpp:3094] Checkpointing ExecutorInfo to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/executor.info'
I1014 17:13:07.358564 61257 slave.cpp:3183] Checkpointing TaskInfo to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/tasks/02786c98-93b9-4a26-ae6d-deefd2708b94/task.info'
I1014 17:13:07.358561 61260 process_isolator.cpp:100] Launching
02786c98-93b9-4a26-ae6d-deefd2708b94
(/home/jyx/versions/mesos2/build/src/mesos-executor) in
/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c
with resources ' for framework 201310141713-2081170186-58876-61234-0000
I1014 17:13:07.358795 61257 slave.cpp:1008] Queuing task
'02786c98-93b9-4a26-ae6d-deefd2708b94' for executor
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
'201310141713-2081170186-58876-61234-0000
I1014 17:13:07.359495 61260 process_isolator.cpp:163] Forked executor at 5601
I1014 17:13:07.359587 61257 slave.cpp:529] Successfully attached file
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c'
Killed the following process trees:
[
--- 5600 sleep 1000
]
Checkpointing executor's forked pid 5601 to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/pids/forked.pid'
Fetching resources into
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c'
Command terminated with signal Killed (pid: 5393)
I1014 17:13:07.413612 5382 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: 7444fe93-fc88-4969-ba39-f00d1f7bfa52) for task
3775fcd5-4776-4e46-aacb-93db531757b1 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.413866 61264 process.cpp:986] Socket closed while receiving
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1014 17:13:07.419656 5601 process.cpp:1555] libprocess is initialized on
10.35.12.124:46774 for 16 cpus
I1014 17:13:07.422185 5645 exec.cpp:174] Executor started at:
executor(1)@10.35.12.124:46774 with pid 5601
I1014 17:13:07.422819 61251 slave.cpp:1460] Got registration for executor
'02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.423228 61251 slave.cpp:1545] Checkpointing executor pid
'executor(1)@10.35.12.124:46774' to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/pids/libprocess.pid'
I1014 17:13:07.424352 5646 process.cpp:986] Socket closed while receiving
I1014 17:13:07.424037 61251 slave.cpp:1581] Flushing queued task
02786c98-93b9-4a26-ae6d-deefd2708b94 for executor
'02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.424475 5635 exec.cpp:198] Executor registered on slave
201310141713-2081170186-58876-61234-0
I1014 17:13:07.424753 5646 process.cpp:986] Socket closed while receiving
I1014 17:13:07.425571 5635 exec.cpp:210] Executor::registered took 101.155us
Registered executor on smfd-aki-27-sr1.devel.twitter.com
I1014 17:13:07.425681 5635 exec.cpp:285] Executor asked to run task
'02786c98-93b9-4a26-ae6d-deefd2708b94'
I1014 17:13:07.425745 5635 exec.cpp:294] Executor::launchTask took 38.183us
Starting task 02786c98-93b9-4a26-ae6d-deefd2708b94
sh -c 'sleep 1000'
Forked command at 5647
I1014 17:13:07.427450 5645 exec.cpp:505] Executor sending status update
TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.427744 61253 slave.cpp:1793] Handling status update TASK_RUNNING
(UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 from executor(1)@10.35.12.124:46774
I1014 17:13:07.427968 61253 status_update_manager.cpp:300] Received status
update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.428227 61253 status_update_manager.cpp:471] Creating
StatusUpdate stream for task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.428529 61253 status_update_manager.hpp:337] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for
task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.530436 61253 status_update_manager.cpp:351] Forwarding status
update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 to master@10.35.12.124:58876
I1014 17:13:07.531108 61253 slave.cpp:1912] Status update manager successfully
handled status update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297)
for task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
../../src/tests/slave_recovery_tests.cpp:1647: Failure
Failed to wait 10secs for status
I1014 17:13:07.531616 61253 slave.cpp:1918] Sending acknowledgement for status
update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 to executor(1)@10.35.12.124:46774
I1014 17:13:07.532291 5646 process.cpp:986] Socket closed while receiving
I1014 17:13:07.532322 5635 exec.cpp:331] Executor received status update
acknowledgement c94ea840-2bc9-40dc-8745-f87a5f0d7297 for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.531469 61256 sched.cpp:527] Scheduler::statusUpdate took 78.668us
I1014 17:13:07.531134 61249 master.cpp:1448] Status update TASK_RUNNING (UUID:
c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 from slave(168)@10.35.12.124:58876
I1014 17:13:07.532776 61255 status_update_manager.cpp:375] Received status
update acknowledgement (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.533468 61255 status_update_manager.hpp:337] Checkpointing ACK
for status update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for
task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.613468 61264 process.cpp:986] Socket closed while receiving
Command terminated with signal Killed (pid: 5440)
I1014 17:13:07.620054 5434 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: 2b2abab7-2036-43c6-8fde-420b862e28df) for task
093ddb89-36cb-45bb-95ad-d2b952536065 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.622478 61255 slave.cpp:1406] Status update manager successfully
handled status update acknowledgement (UUID:
c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.800328 61264 process.cpp:986] Socket closed while receiving
Command terminated with signal Killed (pid: 5487)
I1014 17:13:07.836431 5479 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: 6b714b63-2f0f-4d8c-9e11-03f6173275bf) for task
8c49c32e-a871-47c5-95bd-9a0c200862c1 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.994355 61264 process.cpp:986] Socket closed while receiving
Command terminated with signal Killed (pid: 5534)
I1014 17:13:08.032539 5528 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: cccd6352-21c2-4c82-8b87-f852341bc220) for task
ecdc0322-ae55-4a6d-ac8a-7c060ed791cf of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:08.210850 61264 process.cpp:986] Socket closed while receiving
Command terminated with signal Killed (pid: 5600)
I1014 17:13:08.233747 5583 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: 410c0044-ca6b-4d5f-9278-4a97b23a47c8) for task
12bb99e8-61b2-4f12-91dc-aae39c535360 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:08.349792 61249 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 47.763us
I1014 17:13:08.372403 61259 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
I1014 17:13:08.414371 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:08.621309 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:08.837842 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:09.033443 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:09.235080 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:09.350921 61253 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 26.59us
I1014 17:13:09.384407 61258 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
I1014 17:13:10.352005 61252 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 57.152us
I1014 17:13:10.396427 61258 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
I1014 17:13:11.353577 61250 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 23.812us
I1014 17:13:11.408164 61258 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
I1014 17:13:12.349570 61262 master.cpp:85] No whitelist given. Advertising
offers for all slaves
I1014 17:13:12.355087 61260 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 34.427us
I1014 17:13:12.420876 61253 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
...
was:
sudo MESOS_VERBOSE=1 GLOG_v=1 ./bin/mesos-tests.sh
--gtest_filter="SlaveRecoveryTest/0.RegisterDisconnectedSlave"
--gtest_repeat=1000 --gest_break_on_failure
Error:
../../src/tests/slave_recovery_tests.cpp:1647: Failure
Failed to wait 10secs for status
[ RUN ] SlaveRecoveryTest/0.RegisterDisconnectedSlave
I1014 17:13:07.347805 61261 master.cpp:284] Master started on 10.35.12.124:58876
I1014 17:13:07.347967 61261 master.cpp:299] Master ID:
201310141713-2081170186-58876-61234
I1014 17:13:07.348089 61261 master.cpp:302] Master only allowing authenticated
frameworks to register!
I1014 17:13:07.348464 61259 hierarchical_allocator_process.hpp:302]
Initializing hierarchical allocator process with master :
master@10.35.12.124:58876
I1014 17:13:07.348469 61251 master.cpp:85] No whitelist given. Advertising
offers for all slaves
I1014 17:13:07.348611 61255 slave.cpp:108] Slave started on
168)@10.35.12.124:58876
I1014 17:13:07.348711 61261 master.cpp:697] Elected as master!
I1014 17:13:07.348975 61255 slave.cpp:208] Slave resources: cpus(*):2;
mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I1014 17:13:07.349717 61262 process_isolator.cpp:314] Recovering isolator
I1014 17:13:07.349750 61255 slave.cpp:547] New master detected at
master@10.35.12.124:58876
I1014 17:13:07.350041 61255 slave.cpp:562] Postponing registration until
recovery is complete
I1014 17:13:07.350059 61260 status_update_manager.cpp:157] New master detected
at master@10.35.12.124:58876
I1014 17:13:07.350128 61255 slave.cpp:399] Finished recovery
I1014 17:13:07.350483 61256 master.cpp:1248] Attempting to register slave on
smfd-aki-27-sr1.devel.twitter.com at slave(168)@10.35.12.124:58876
I1014 17:13:07.350579 61256 master.cpp:2502] Adding slave
201310141713-2081170186-58876-61234-0 at smfd-aki-27-sr1.devel.twitter.com with
cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I1014 17:13:07.350703 61258 slave.cpp:613] Registered with master
master@10.35.12.124:58876; given slave ID 201310141713-2081170186-58876-61234-0
I1014 17:13:07.350824 61250 hierarchical_allocator_process.hpp:445] Added slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com) with
cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2;
mem(*):1024; disk(*):1024; ports(*):[31000-32000] available)
I1014 17:13:07.350941 61258 slave.cpp:626] Checkpointing SlaveInfo to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/slave.info'
I1014 17:13:07.350998 61250 hierarchical_allocator_process.hpp:708] Performed
allocation for slave 201310141713-2081170186-58876-61234-0 in 7.03us
I1014 17:13:07.351054 61254 sched.cpp:195] New master at
master@10.35.12.124:58876
I1014 17:13:07.351310 61254 sched.cpp:281] Authenticating with master
master@10.35.12.124:58876
I1014 17:13:07.351511 61263 authenticatee.hpp:124] Creating new client SASL
connection
I1014 17:13:07.351646 61253 master.cpp:1723] Authenticating framework at
scheduler(168)@10.35.12.124:58876
I1014 17:13:07.351886 61257 authenticator.hpp:140] Creating new server SASL
connection
I1014 17:13:07.352033 61253 authenticatee.hpp:212] Received SASL authentication
mechanisms: CRAM-MD5
I1014 17:13:07.352105 61253 authenticatee.hpp:238] Attempting to authenticate
with mechanism 'CRAM-MD5'
I1014 17:13:07.352202 61255 authenticator.hpp:243] Received SASL authentication
start
I1014 17:13:07.352306 61255 authenticator.hpp:325] Authentication requires more
steps
I1014 17:13:07.352394 61255 authenticatee.hpp:258] Received SASL authentication
step
I1014 17:13:07.352504 61255 authenticator.hpp:271] Received SASL authentication
step
I1014 17:13:07.352581 61255 auxprop.cpp:81] Request to lookup properties for
user: 'test-principal' realm: 'smfd-aki-27-sr1.devel.twitter.com' server FQDN:
'smfd-aki-27-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false
SASL_AUXPROP_AUTHZID: false
I1014 17:13:07.352654 61255 auxprop.cpp:153] Looking up auxiliary property
'*userPassword'
I1014 17:13:07.352728 61255 auxprop.cpp:153] Looking up auxiliary property
'*cmusaslsecretCRAM-MD5'
I1014 17:13:07.352825 61255 auxprop.cpp:81] Request to lookup properties for
user: 'test-principal' realm: 'smfd-aki-27-sr1.devel.twitter.com' server FQDN:
'smfd-aki-27-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false
SASL_AUXPROP_AUTHZID: true
I1014 17:13:07.352893 61255 auxprop.cpp:103] Skipping auxiliary property
'*userPassword' since SASL_AUXPROP_AUTHZID == true
I1014 17:13:07.352957 61255 auxprop.cpp:103] Skipping auxiliary property
'*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1014 17:13:07.353035 61255 authenticator.hpp:317] Authentication success
I1014 17:13:07.353150 61253 authenticatee.hpp:298] Authentication success
I1014 17:13:07.353183 61254 master.cpp:1763] Successfully authenticated
framework at scheduler(168)@10.35.12.124:58876
I1014 17:13:07.353281 61253 sched.cpp:326] Successfully authenticated with
master master@10.35.12.124:58876
I1014 17:13:07.353430 61254 master.cpp:768] Received registration request from
scheduler(168)@10.35.12.124:58876
I1014 17:13:07.353574 61254 master.cpp:786] Registering framework
201310141713-2081170186-58876-61234-0000 at scheduler(168)@10.35.12.124:58876
I1014 17:13:07.353706 61259 sched.cpp:365] Framework registered with
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.353735 61255 hierarchical_allocator_process.hpp:332] Added
framework 201310141713-2081170186-58876-61234-0000
I1014 17:13:07.353799 61259 sched.cpp:379] Scheduler::registered took 9.296us
I1014 17:13:07.353881 61255 hierarchical_allocator_process.hpp:752] Offering
cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 to framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.354061 61255 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 212.58us
I1014 17:13:07.354136 61262 master.hpp:389] Adding offer
201310141713-2081170186-58876-61234-0 with resources cpus(*):2; mem(*):1024;
disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
I1014 17:13:07.354249 61262 master.cpp:1689] Sending 1 offers to framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.354419 61260 sched.cpp:472] Scheduler::resourceOffers took
22.071us
I1014 17:13:07.355800 61250 master.cpp:2015] Processing reply for offer
201310141713-2081170186-58876-61234-0 on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com) for
framework 201310141713-2081170186-58876-61234-0000
I1014 17:13:07.355950 61250 master.hpp:361] Adding task
02786c98-93b9-4a26-ae6d-deefd2708b94 with resources cpus(*):2; mem(*):1024;
disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
I1014 17:13:07.356057 61250 master.cpp:2139] Launching task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 with resources cpus(*):2; mem(*):1024;
disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
I1014 17:13:07.356216 61257 slave.cpp:786] Got assigned task
02786c98-93b9-4a26-ae6d-deefd2708b94 for framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.356230 61250 master.hpp:399] Removing offer
201310141713-2081170186-58876-61234-0 with resources cpus(*):2; mem(*):1024;
disk(*):1024; ports(*):[31000-32000] on slave
201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
I1014 17:13:07.356366 61257 slave.cpp:2844] Checkpointing FrameworkInfo to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/framework.info'
I1014 17:13:07.356609 61257 slave.cpp:2851] Checkpointing framework pid
'scheduler(168)@10.35.12.124:58876' to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/framework.pid'
I1014 17:13:07.356909 61257 slave.cpp:897] Launching task
02786c98-93b9-4a26-ae6d-deefd2708b94 for framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.358155 61257 slave.cpp:3094] Checkpointing ExecutorInfo to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/executor.info'
I1014 17:13:07.358564 61257 slave.cpp:3183] Checkpointing TaskInfo to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/tasks/02786c98-93b9-4a26-ae6d-deefd2708b94/task.info'
I1014 17:13:07.358561 61260 process_isolator.cpp:100] Launching
02786c98-93b9-4a26-ae6d-deefd2708b94
(/home/jyx/versions/mesos2/build/src/mesos-executor) in
/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c
with resources ' for framework 201310141713-2081170186-58876-61234-0000
I1014 17:13:07.358795 61257 slave.cpp:1008] Queuing task
'02786c98-93b9-4a26-ae6d-deefd2708b94' for executor
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
'201310141713-2081170186-58876-61234-0000
I1014 17:13:07.359495 61260 process_isolator.cpp:163] Forked executor at 5601
I1014 17:13:07.359587 61257 slave.cpp:529] Successfully attached file
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c'
Killed the following process trees:
[
--- 5600 sleep 1000
]
Checkpointing executor's forked pid 5601 to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/pids/forked.pid'
Fetching resources into
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c'
Command terminated with signal Killed (pid: 5393)
I1014 17:13:07.413612 5382 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: 7444fe93-fc88-4969-ba39-f00d1f7bfa52) for task
3775fcd5-4776-4e46-aacb-93db531757b1 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.413866 61264 process.cpp:986] Socket closed while receiving
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1014 17:13:07.419656 5601 process.cpp:1555] libprocess is initialized on
10.35.12.124:46774 for 16 cpus
I1014 17:13:07.422185 5645 exec.cpp:174] Executor started at:
executor(1)@10.35.12.124:46774 with pid 5601
I1014 17:13:07.422819 61251 slave.cpp:1460] Got registration for executor
'02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.423228 61251 slave.cpp:1545] Checkpointing executor pid
'executor(1)@10.35.12.124:46774' to
'/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/pids/libprocess.pid'
I1014 17:13:07.424352 5646 process.cpp:986] Socket closed while receiving
I1014 17:13:07.424037 61251 slave.cpp:1581] Flushing queued task
02786c98-93b9-4a26-ae6d-deefd2708b94 for executor
'02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.424475 5635 exec.cpp:198] Executor registered on slave
201310141713-2081170186-58876-61234-0
I1014 17:13:07.424753 5646 process.cpp:986] Socket closed while receiving
I1014 17:13:07.425571 5635 exec.cpp:210] Executor::registered took 101.155us
Registered executor on smfd-aki-27-sr1.devel.twitter.com
I1014 17:13:07.425681 5635 exec.cpp:285] Executor asked to run task
'02786c98-93b9-4a26-ae6d-deefd2708b94'
I1014 17:13:07.425745 5635 exec.cpp:294] Executor::launchTask took 38.183us
Starting task 02786c98-93b9-4a26-ae6d-deefd2708b94
sh -c 'sleep 1000'
Forked command at 5647
I1014 17:13:07.427450 5645 exec.cpp:505] Executor sending status update
TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.427744 61253 slave.cpp:1793] Handling status update TASK_RUNNING
(UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 from executor(1)@10.35.12.124:46774
I1014 17:13:07.427968 61253 status_update_manager.cpp:300] Received status
update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.428227 61253 status_update_manager.cpp:471] Creating
StatusUpdate stream for task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.428529 61253 status_update_manager.hpp:337] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for
task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.530436 61253 status_update_manager.cpp:351] Forwarding status
update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 to master@10.35.12.124:58876
I1014 17:13:07.531108 61253 slave.cpp:1912] Status update manager successfully
handled status update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297)
for task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
../../src/tests/slave_recovery_tests.cpp:1647: Failure
Failed to wait 10secs for status
I1014 17:13:07.531616 61253 slave.cpp:1918] Sending acknowledgement for status
update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 to executor(1)@10.35.12.124:46774
I1014 17:13:07.532291 5646 process.cpp:986] Socket closed while receiving
I1014 17:13:07.532322 5635 exec.cpp:331] Executor received status update
acknowledgement c94ea840-2bc9-40dc-8745-f87a5f0d7297 for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.531469 61256 sched.cpp:527] Scheduler::statusUpdate took 78.668us
I1014 17:13:07.531134 61249 master.cpp:1448] Status update TASK_RUNNING (UUID:
c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000 from slave(168)@10.35.12.124:58876
I1014 17:13:07.532776 61255 status_update_manager.cpp:375] Received status
update acknowledgement (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.533468 61255 status_update_manager.hpp:337] Checkpointing ACK
for status update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for
task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.613468 61264 process.cpp:986] Socket closed while receiving
Command terminated with signal Killed (pid: 5440)
I1014 17:13:07.620054 5434 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: 2b2abab7-2036-43c6-8fde-420b862e28df) for task
093ddb89-36cb-45bb-95ad-d2b952536065 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.622478 61255 slave.cpp:1406] Status update manager successfully
handled status update acknowledgement (UUID:
c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.800328 61264 process.cpp:986] Socket closed while receiving
Command terminated with signal Killed (pid: 5487)
I1014 17:13:07.836431 5479 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: 6b714b63-2f0f-4d8c-9e11-03f6173275bf) for task
8c49c32e-a871-47c5-95bd-9a0c200862c1 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:07.994355 61264 process.cpp:986] Socket closed while receiving
Command terminated with signal Killed (pid: 5534)
I1014 17:13:08.032539 5528 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: cccd6352-21c2-4c82-8b87-f852341bc220) for task
ecdc0322-ae55-4a6d-ac8a-7c060ed791cf of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:08.210850 61264 process.cpp:986] Socket closed while receiving
Command terminated with signal Killed (pid: 5600)
I1014 17:13:08.233747 5583 exec.cpp:505] Executor sending status update
TASK_KILLED (UUID: 410c0044-ca6b-4d5f-9278-4a97b23a47c8) for task
12bb99e8-61b2-4f12-91dc-aae39c535360 of framework
201310141713-2081170186-58876-61234-0000
I1014 17:13:08.349792 61249 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 47.763us
I1014 17:13:08.372403 61259 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
I1014 17:13:08.414371 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:08.621309 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:08.837842 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:09.033443 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:09.235080 61264 process.cpp:986] Socket closed while receiving
I1014 17:13:09.350921 61253 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 26.59us
I1014 17:13:09.384407 61258 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
I1014 17:13:10.352005 61252 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 57.152us
I1014 17:13:10.396427 61258 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
I1014 17:13:11.353577 61250 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 23.812us
I1014 17:13:11.408164 61258 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
I1014 17:13:12.349570 61262 master.cpp:85] No whitelist given. Advertising
offers for all slaves
I1014 17:13:12.355087 61260 hierarchical_allocator_process.hpp:688] Performed
allocation for 1 slaves in 34.427us
I1014 17:13:12.420876 61253 monitor.cpp:187] Publishing resource usage for
executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
'201310141713-2081170186-58876-61234-0000'
...
> SlaveRecoveryTest/0.RegisterDisconnectedSlave flaky
> ---------------------------------------------------
>
> Key: MESOS-735
> URL: https://issues.apache.org/jira/browse/MESOS-735
> Project: Mesos
> Issue Type: Bug
> Reporter: Yan Xu
>
> sudo MESOS_VERBOSE=1 GLOG_v=1 ./bin/mesos-tests.sh
> --gtest_filter="SlaveRecoveryTest/0.RegisterDisconnectedSlave"
> --gtest_repeat=1000 --gest_break_on_failure
> Error:
> ../../src/tests/slave_recovery_tests.cpp:1647: Failure
> Failed to wait 10secs for status
> The TASK_KILLED status for task "3775fcd5-4776-4e46-aacb-93db531757b1" is
> from a previous iteration of the test.
> Log:
> [ RUN ] SlaveRecoveryTest/0.RegisterDisconnectedSlave
> I1014 17:13:07.347805 61261 master.cpp:284] Master started on
> 10.35.12.124:58876
> I1014 17:13:07.347967 61261 master.cpp:299] Master ID:
> 201310141713-2081170186-58876-61234
> I1014 17:13:07.348089 61261 master.cpp:302] Master only allowing
> authenticated frameworks to register!
> I1014 17:13:07.348464 61259 hierarchical_allocator_process.hpp:302]
> Initializing hierarchical allocator process with master :
> master@10.35.12.124:58876
> I1014 17:13:07.348469 61251 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 17:13:07.348611 61255 slave.cpp:108] Slave started on
> 168)@10.35.12.124:58876
> I1014 17:13:07.348711 61261 master.cpp:697] Elected as master!
> I1014 17:13:07.348975 61255 slave.cpp:208] Slave resources: cpus(*):2;
> mem(*):1024; disk(*):1024; ports(*):[31000-32000]
> I1014 17:13:07.349717 61262 process_isolator.cpp:314] Recovering isolator
> I1014 17:13:07.349750 61255 slave.cpp:547] New master detected at
> master@10.35.12.124:58876
> I1014 17:13:07.350041 61255 slave.cpp:562] Postponing registration until
> recovery is complete
> I1014 17:13:07.350059 61260 status_update_manager.cpp:157] New master
> detected at master@10.35.12.124:58876
> I1014 17:13:07.350128 61255 slave.cpp:399] Finished recovery
> I1014 17:13:07.350483 61256 master.cpp:1248] Attempting to register slave on
> smfd-aki-27-sr1.devel.twitter.com at slave(168)@10.35.12.124:58876
> I1014 17:13:07.350579 61256 master.cpp:2502] Adding slave
> 201310141713-2081170186-58876-61234-0 at smfd-aki-27-sr1.devel.twitter.com
> with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
> I1014 17:13:07.350703 61258 slave.cpp:613] Registered with master
> master@10.35.12.124:58876; given slave ID
> 201310141713-2081170186-58876-61234-0
> I1014 17:13:07.350824 61250 hierarchical_allocator_process.hpp:445] Added
> slave 201310141713-2081170186-58876-61234-0
> (smfd-aki-27-sr1.devel.twitter.com) with cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000] available)
> I1014 17:13:07.350941 61258 slave.cpp:626] Checkpointing SlaveInfo to
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/slave.info'
> I1014 17:13:07.350998 61250 hierarchical_allocator_process.hpp:708] Performed
> allocation for slave 201310141713-2081170186-58876-61234-0 in 7.03us
> I1014 17:13:07.351054 61254 sched.cpp:195] New master at
> master@10.35.12.124:58876
> I1014 17:13:07.351310 61254 sched.cpp:281] Authenticating with master
> master@10.35.12.124:58876
> I1014 17:13:07.351511 61263 authenticatee.hpp:124] Creating new client SASL
> connection
> I1014 17:13:07.351646 61253 master.cpp:1723] Authenticating framework at
> scheduler(168)@10.35.12.124:58876
> I1014 17:13:07.351886 61257 authenticator.hpp:140] Creating new server SASL
> connection
> I1014 17:13:07.352033 61253 authenticatee.hpp:212] Received SASL
> authentication mechanisms: CRAM-MD5
> I1014 17:13:07.352105 61253 authenticatee.hpp:238] Attempting to authenticate
> with mechanism 'CRAM-MD5'
> I1014 17:13:07.352202 61255 authenticator.hpp:243] Received SASL
> authentication start
> I1014 17:13:07.352306 61255 authenticator.hpp:325] Authentication requires
> more steps
> I1014 17:13:07.352394 61255 authenticatee.hpp:258] Received SASL
> authentication step
> I1014 17:13:07.352504 61255 authenticator.hpp:271] Received SASL
> authentication step
> I1014 17:13:07.352581 61255 auxprop.cpp:81] Request to lookup properties for
> user: 'test-principal' realm: 'smfd-aki-27-sr1.devel.twitter.com' server
> FQDN: 'smfd-aki-27-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false
> SASL_AUXPROP_AUTHZID: false
> I1014 17:13:07.352654 61255 auxprop.cpp:153] Looking up auxiliary property
> '*userPassword'
> I1014 17:13:07.352728 61255 auxprop.cpp:153] Looking up auxiliary property
> '*cmusaslsecretCRAM-MD5'
> I1014 17:13:07.352825 61255 auxprop.cpp:81] Request to lookup properties for
> user: 'test-principal' realm: 'smfd-aki-27-sr1.devel.twitter.com' server
> FQDN: 'smfd-aki-27-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false
> SASL_AUXPROP_AUTHZID: true
> I1014 17:13:07.352893 61255 auxprop.cpp:103] Skipping auxiliary property
> '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I1014 17:13:07.352957 61255 auxprop.cpp:103] Skipping auxiliary property
> '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I1014 17:13:07.353035 61255 authenticator.hpp:317] Authentication success
> I1014 17:13:07.353150 61253 authenticatee.hpp:298] Authentication success
> I1014 17:13:07.353183 61254 master.cpp:1763] Successfully authenticated
> framework at scheduler(168)@10.35.12.124:58876
> I1014 17:13:07.353281 61253 sched.cpp:326] Successfully authenticated with
> master master@10.35.12.124:58876
> I1014 17:13:07.353430 61254 master.cpp:768] Received registration request
> from scheduler(168)@10.35.12.124:58876
> I1014 17:13:07.353574 61254 master.cpp:786] Registering framework
> 201310141713-2081170186-58876-61234-0000 at scheduler(168)@10.35.12.124:58876
> I1014 17:13:07.353706 61259 sched.cpp:365] Framework registered with
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.353735 61255 hierarchical_allocator_process.hpp:332] Added
> framework 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.353799 61259 sched.cpp:379] Scheduler::registered took 9.296us
> I1014 17:13:07.353881 61255 hierarchical_allocator_process.hpp:752] Offering
> cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave
> 201310141713-2081170186-58876-61234-0 to framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.354061 61255 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 212.58us
> I1014 17:13:07.354136 61262 master.hpp:389] Adding offer
> 201310141713-2081170186-58876-61234-0 with resources cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000] on slave
> 201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1014 17:13:07.354249 61262 master.cpp:1689] Sending 1 offers to framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.354419 61260 sched.cpp:472] Scheduler::resourceOffers took
> 22.071us
> I1014 17:13:07.355800 61250 master.cpp:2015] Processing reply for offer
> 201310141713-2081170186-58876-61234-0 on slave
> 201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com) for
> framework 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.355950 61250 master.hpp:361] Adding task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 with resources cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000] on slave
> 201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1014 17:13:07.356057 61250 master.cpp:2139] Launching task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000 with resources cpus(*):2;
> mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave
> 201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1014 17:13:07.356216 61257 slave.cpp:786] Got assigned task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 for framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.356230 61250 master.hpp:399] Removing offer
> 201310141713-2081170186-58876-61234-0 with resources cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000] on slave
> 201310141713-2081170186-58876-61234-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1014 17:13:07.356366 61257 slave.cpp:2844] Checkpointing FrameworkInfo to
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/framework.info'
> I1014 17:13:07.356609 61257 slave.cpp:2851] Checkpointing framework pid
> 'scheduler(168)@10.35.12.124:58876' to
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/framework.pid'
> I1014 17:13:07.356909 61257 slave.cpp:897] Launching task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 for framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.358155 61257 slave.cpp:3094] Checkpointing ExecutorInfo to
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/executor.info'
> I1014 17:13:07.358564 61257 slave.cpp:3183] Checkpointing TaskInfo to
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/tasks/02786c98-93b9-4a26-ae6d-deefd2708b94/task.info'
> I1014 17:13:07.358561 61260 process_isolator.cpp:100] Launching
> 02786c98-93b9-4a26-ae6d-deefd2708b94
> (/home/jyx/versions/mesos2/build/src/mesos-executor) in
> /tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c
> with resources ' for framework 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.358795 61257 slave.cpp:1008] Queuing task
> '02786c98-93b9-4a26-ae6d-deefd2708b94' for executor
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> '201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.359495 61260 process_isolator.cpp:163] Forked executor at 5601
> I1014 17:13:07.359587 61257 slave.cpp:529] Successfully attached file
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c'
> Killed the following process trees:
> [
> --- 5600 sleep 1000
> ]
> Checkpointing executor's forked pid 5601 to
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/pids/forked.pid'
> Fetching resources into
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c'
> Command terminated with signal Killed (pid: 5393)
> I1014 17:13:07.413612 5382 exec.cpp:505] Executor sending status update
> TASK_KILLED (UUID: 7444fe93-fc88-4969-ba39-f00d1f7bfa52) for task
> 3775fcd5-4776-4e46-aacb-93db531757b1 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.413866 61264 process.cpp:986] Socket closed while receiving
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1014 17:13:07.419656 5601 process.cpp:1555] libprocess is initialized on
> 10.35.12.124:46774 for 16 cpus
> I1014 17:13:07.422185 5645 exec.cpp:174] Executor started at:
> executor(1)@10.35.12.124:46774 with pid 5601
> I1014 17:13:07.422819 61251 slave.cpp:1460] Got registration for executor
> '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.423228 61251 slave.cpp:1545] Checkpointing executor pid
> 'executor(1)@10.35.12.124:46774' to
> '/tmp/SlaveRecoveryTest_0_RegisterDisconnectedSlave_ne8Shz/meta/slaves/201310141713-2081170186-58876-61234-0/frameworks/201310141713-2081170186-58876-61234-0000/executors/02786c98-93b9-4a26-ae6d-deefd2708b94/runs/f6cb0202-06c2-47d2-838c-1a2d9d9e301c/pids/libprocess.pid'
> I1014 17:13:07.424352 5646 process.cpp:986] Socket closed while receiving
> I1014 17:13:07.424037 61251 slave.cpp:1581] Flushing queued task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 for executor
> '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.424475 5635 exec.cpp:198] Executor registered on slave
> 201310141713-2081170186-58876-61234-0
> I1014 17:13:07.424753 5646 process.cpp:986] Socket closed while receiving
> I1014 17:13:07.425571 5635 exec.cpp:210] Executor::registered took 101.155us
> Registered executor on smfd-aki-27-sr1.devel.twitter.com
> I1014 17:13:07.425681 5635 exec.cpp:285] Executor asked to run task
> '02786c98-93b9-4a26-ae6d-deefd2708b94'
> I1014 17:13:07.425745 5635 exec.cpp:294] Executor::launchTask took 38.183us
> Starting task 02786c98-93b9-4a26-ae6d-deefd2708b94
> sh -c 'sleep 1000'
> Forked command at 5647
> I1014 17:13:07.427450 5645 exec.cpp:505] Executor sending status update
> TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.427744 61253 slave.cpp:1793] Handling status update
> TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000 from executor(1)@10.35.12.124:46774
> I1014 17:13:07.427968 61253 status_update_manager.cpp:300] Received status
> update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.428227 61253 status_update_manager.cpp:471] Creating
> StatusUpdate stream for task 02786c98-93b9-4a26-ae6d-deefd2708b94 of
> framework 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.428529 61253 status_update_manager.hpp:337] Checkpointing
> UPDATE for status update TASK_RUNNING (UUID:
> c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.530436 61253 status_update_manager.cpp:351] Forwarding status
> update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000 to master@10.35.12.124:58876
> I1014 17:13:07.531108 61253 slave.cpp:1912] Status update manager
> successfully handled status update TASK_RUNNING (UUID:
> c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000
> ../../src/tests/slave_recovery_tests.cpp:1647: Failure
> Failed to wait 10secs for status
> I1014 17:13:07.531616 61253 slave.cpp:1918] Sending acknowledgement for
> status update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for
> task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000 to executor(1)@10.35.12.124:46774
> I1014 17:13:07.532291 5646 process.cpp:986] Socket closed while receiving
> I1014 17:13:07.532322 5635 exec.cpp:331] Executor received status update
> acknowledgement c94ea840-2bc9-40dc-8745-f87a5f0d7297 for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.531469 61256 sched.cpp:527] Scheduler::statusUpdate took
> 78.668us
> I1014 17:13:07.531134 61249 master.cpp:1448] Status update TASK_RUNNING
> (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000 from slave(168)@10.35.12.124:58876
> I1014 17:13:07.532776 61255 status_update_manager.cpp:375] Received status
> update acknowledgement (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.533468 61255 status_update_manager.hpp:337] Checkpointing ACK
> for status update TASK_RUNNING (UUID: c94ea840-2bc9-40dc-8745-f87a5f0d7297)
> for task 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.613468 61264 process.cpp:986] Socket closed while receiving
> Command terminated with signal Killed (pid: 5440)
> I1014 17:13:07.620054 5434 exec.cpp:505] Executor sending status update
> TASK_KILLED (UUID: 2b2abab7-2036-43c6-8fde-420b862e28df) for task
> 093ddb89-36cb-45bb-95ad-d2b952536065 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.622478 61255 slave.cpp:1406] Status update manager
> successfully handled status update acknowledgement (UUID:
> c94ea840-2bc9-40dc-8745-f87a5f0d7297) for task
> 02786c98-93b9-4a26-ae6d-deefd2708b94 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.800328 61264 process.cpp:986] Socket closed while receiving
> Command terminated with signal Killed (pid: 5487)
> I1014 17:13:07.836431 5479 exec.cpp:505] Executor sending status update
> TASK_KILLED (UUID: 6b714b63-2f0f-4d8c-9e11-03f6173275bf) for task
> 8c49c32e-a871-47c5-95bd-9a0c200862c1 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:07.994355 61264 process.cpp:986] Socket closed while receiving
> Command terminated with signal Killed (pid: 5534)
> I1014 17:13:08.032539 5528 exec.cpp:505] Executor sending status update
> TASK_KILLED (UUID: cccd6352-21c2-4c82-8b87-f852341bc220) for task
> ecdc0322-ae55-4a6d-ac8a-7c060ed791cf of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:08.210850 61264 process.cpp:986] Socket closed while receiving
> Command terminated with signal Killed (pid: 5600)
> I1014 17:13:08.233747 5583 exec.cpp:505] Executor sending status update
> TASK_KILLED (UUID: 410c0044-ca6b-4d5f-9278-4a97b23a47c8) for task
> 12bb99e8-61b2-4f12-91dc-aae39c535360 of framework
> 201310141713-2081170186-58876-61234-0000
> I1014 17:13:08.349792 61249 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 47.763us
> I1014 17:13:08.372403 61259 monitor.cpp:187] Publishing resource usage for
> executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
> '201310141713-2081170186-58876-61234-0000'
> I1014 17:13:08.414371 61264 process.cpp:986] Socket closed while receiving
> I1014 17:13:08.621309 61264 process.cpp:986] Socket closed while receiving
> I1014 17:13:08.837842 61264 process.cpp:986] Socket closed while receiving
> I1014 17:13:09.033443 61264 process.cpp:986] Socket closed while receiving
> I1014 17:13:09.235080 61264 process.cpp:986] Socket closed while receiving
> I1014 17:13:09.350921 61253 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 26.59us
> I1014 17:13:09.384407 61258 monitor.cpp:187] Publishing resource usage for
> executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
> '201310141713-2081170186-58876-61234-0000'
> I1014 17:13:10.352005 61252 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 57.152us
> I1014 17:13:10.396427 61258 monitor.cpp:187] Publishing resource usage for
> executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
> '201310141713-2081170186-58876-61234-0000'
> I1014 17:13:11.353577 61250 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 23.812us
> I1014 17:13:11.408164 61258 monitor.cpp:187] Publishing resource usage for
> executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
> '201310141713-2081170186-58876-61234-0000'
> I1014 17:13:12.349570 61262 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 17:13:12.355087 61260 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 34.427us
> I1014 17:13:12.420876 61253 monitor.cpp:187] Publishing resource usage for
> executor '02786c98-93b9-4a26-ae6d-deefd2708b94' of framework
> '201310141713-2081170186-58876-61234-0000'
> ...
--
This message was sent by Atlassian JIRA
(v6.1#6144)
