[ https://issues.apache.org/jira/browse/MESOS-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kone updated MESOS-738: ----------------------------- Fix Version/s: 0.15.0 Assignee: Vinod Kone https://reviews.apache.org/r/14640/ > CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework_NoBuffer can't finish > ----------------------------------------------------------------------- > > Key: MESOS-738 > URL: https://issues.apache.org/jira/browse/MESOS-738 > Project: Mesos > Issue Type: Bug > Reporter: Yan Xu > Assignee: Vinod Kone > Fix For: 0.15.0 > > > This doesn't seem to be a problem on Jenkins. Wondering what was causing the > problem on my test box... > Test command: > sudo MESOS_VERBOSE=1 GLOG_v=1 make check -j > GTEST_FILTER="CgroupsIsolatorTest/ROOT_CGROUPS_BalloonFramework*" > Log: > [ RUN ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework_NoBuffer > Using temporary directory > '/tmp/CgroupsIsolatorTest_ROOT_CGROUPS_BalloonFramework_NoBuffer_UkYn2q' > Launched master at 56065 > WARNING: Logging before InitGoogleLogging() is written to STDERR > I1014 21:56:49.593399 56065 process.cpp:1555] libprocess is initialized on > 127.0.0.1:5432 for 24 cpus > I1014 21:56:49.593868 56065 logging.cpp:110] Logging to STDERR > I1014 21:56:49.594128 56065 main.cpp:114] Build: 2013-10-14 18:18:08 by root > I1014 21:56:49.594156 56065 main.cpp:115] Starting Mesos master > I1014 21:56:49.594457 56084 master.cpp:284] Master started on 127.0.0.1:5432 > I1014 21:56:49.594609 56084 master.cpp:299] Master ID: > 201310142156-16777343-5432-56065 > I1014 21:56:49.594671 56084 master.cpp:302] Master only allowing > authenticated frameworks to register! > I1014 21:56:49.595432 56101 hierarchical_allocator_process.hpp:302] > Initializing hierarchical allocator process with master : > master@127.0.0.1:5432 > I1014 21:56:49.595450 56090 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:56:49.596568 56084 master.cpp:697] Elected as master! > I1014 21:56:50.595949 56086 hierarchical_allocator_process.hpp:726] No > resources available to allocate! > I1014 21:56:50.596043 56086 hierarchical_allocator_process.hpp:688] Performed > allocation for 0 slaves in 108.976us > Launched slave at 56114 > I1014 21:56:51.596371 56089 hierarchical_allocator_process.hpp:726] No > resources available to allocate! > I1014 21:56:51.596426 56089 hierarchical_allocator_process.hpp:688] Performed > allocation for 0 slaves in 66.034us > WARNING: Logging before InitGoogleLogging() is written to STDERR > I1014 21:56:51.600865 56114 process.cpp:1555] libprocess is initialized on > 10.35.255.108:5051 for 24 cpus > I1014 21:56:51.601160 56114 logging.cpp:110] Logging to STDERR > I1014 21:56:51.601503 56114 main.cpp:119] Creating "cgroups" isolator > I1014 21:56:51.601678 56114 main.cpp:127] Build: 2013-10-14 18:18:08 by root > I1014 21:56:51.601706 56114 main.cpp:128] Starting Mesos slave > I1014 21:56:51.603461 56137 slave.cpp:108] Slave started on > 1)@10.35.255.108:5051 > I1014 21:56:51.604014 56137 slave.cpp:208] Slave resources: cpus(*):1; > mem(*):96; disk(*):454895; ports(*):[31000-32000] > I1014 21:56:51.605553 56138 cgroups_isolator.cpp:224] Using > /tmp/mesos_test_cgroup as cgroups hierarchy root > I1014 21:56:51.606158 56137 slave.cpp:547] New master detected at > master@127.0.0.1:5432 > I1014 21:56:51.606353 56139 status_update_manager.cpp:157] New master > detected at master@127.0.0.1:5432 > I1014 21:56:51.606314 56137 slave.cpp:562] Postponing registration until > recovery is complete > I1014 21:56:52.597699 56086 hierarchical_allocator_process.hpp:726] No > resources available to allocate! > I1014 21:56:52.597754 56086 hierarchical_allocator_process.hpp:688] Performed > allocation for 0 slaves in 64.23us > I1014 21:56:52.604190 56138 cgroups_isolator.cpp:817] Recovering isolator > I1014 21:56:52.604665 56138 slave.cpp:399] Finished recovery > I1014 21:56:52.605499 56089 master.cpp:1248] Attempting to register slave on > smfd-atr-11-sr1.devel.twitter.com at slave(1)@10.35.255.108:5051 > I1014 21:56:52.605533 56089 master.cpp:2502] Adding slave > 201310142156-16777343-5432-56065-0 at smfd-atr-11-sr1.devel.twitter.com with > cpus(*):1; mem(*):96; disk(*):454895; ports(*):[31000-32000] > I1014 21:56:52.605934 56080 hierarchical_allocator_process.hpp:445] Added > slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) > with cpus(*):1; mem(*):96; disk(*):454895; ports(*):[31000-32000] (and > cpus(*):1; mem(*):96; disk(*):454895; ports(*):[31000-32000] available) > I1014 21:56:52.606008 56080 hierarchical_allocator_process.hpp:708] Performed > allocation for slave 201310142156-16777343-5432-56065-0 in 17.439us > I1014 21:56:52.605998 56146 slave.cpp:613] Registered with master > master@127.0.0.1:5432; given slave ID 201310142156-16777343-5432-56065-0 > Enabling authentication for the framework > WARNING: Logging before InitGoogleLogging() is written to STDERR > I1014 21:56:53.598536 56084 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 25.985us > I1014 21:56:53.598472 56159 process.cpp:1555] libprocess is initialized on > 10.35.255.108:45654 for 24 cpus > I1014 21:56:53.599010 56159 logging.cpp:110] Logging to STDERR > I1014 21:56:53.601263 56182 sched.cpp:195] New master at master@127.0.0.1:5432 > I1014 21:56:53.601438 56182 sched.cpp:281] Authenticating with master > master@127.0.0.1:5432 > I1014 21:56:53.601858 56186 authenticatee.hpp:100] Initializing client SASL > I1014 21:56:53.603885 56186 authenticatee.hpp:124] Creating new client SASL > connection > I1014 21:56:53.604517 56089 master.cpp:1723] Authenticating framework at > scheduler(1)@10.35.255.108:45654 > I1014 21:56:53.604909 56102 authenticator.hpp:83] Initializing server SASL > I1014 21:56:53.606436 56102 auxprop.cpp:45] Initialized in-memory auxiliary > property plugin > I1014 21:56:53.606469 56102 authenticator.hpp:140] Creating new server SASL > connection > I1014 21:56:53.606798 56182 authenticatee.hpp:212] Received SASL > authentication mechanisms: CRAM-MD5 > I1014 21:56:53.606838 56182 authenticatee.hpp:238] Attempting to authenticate > with mechanism 'CRAM-MD5' > I1014 21:56:53.607024 56102 authenticator.hpp:243] Received SASL > authentication start > I1014 21:56:53.607133 56102 authenticator.hpp:325] Authentication requires > more steps > I1014 21:56:53.607283 56191 authenticatee.hpp:258] Received SASL > authentication step > I1014 21:56:53.607574 56096 authenticator.hpp:271] Received SASL > authentication step > I1014 21:56:53.607641 56096 auxprop.cpp:81] Request to lookup properties for > user: 'test-principal' realm: 'smfd-atr-11-sr1.devel.twitter.com' server > FQDN: 'smfd-atr-11-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false > SASL_AUXPROP_AUTHZID: false > I1014 21:56:53.607661 56096 auxprop.cpp:153] Looking up auxiliary property > '*userPassword' > I1014 21:56:53.607683 56096 auxprop.cpp:153] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I1014 21:56:53.607699 56096 auxprop.cpp:81] Request to lookup properties for > user: 'test-principal' realm: 'smfd-atr-11-sr1.devel.twitter.com' server > FQDN: 'smfd-atr-11-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false > SASL_AUXPROP_AUTHZID: true > I1014 21:56:53.607712 56096 auxprop.cpp:103] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I1014 21:56:53.607724 56096 auxprop.cpp:103] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I1014 21:56:53.607745 56096 authenticator.hpp:317] Authentication success > I1014 21:56:53.608036 56192 authenticatee.hpp:298] Authentication success > I1014 21:56:53.608242 56196 sched.cpp:326] Successfully authenticated with > master master@127.0.0.1:5432 > I1014 21:56:53.607980 56093 master.cpp:1763] Successfully authenticated > framework at scheduler(1)@10.35.255.108:45654 > I1014 21:56:53.608603 56093 master.cpp:768] Received registration request > from scheduler(1)@10.35.255.108:45654 > I1014 21:56:53.608747 56093 master.cpp:786] Registering framework > 201310142156-16777343-5432-56065-0000 at scheduler(1)@10.35.255.108:45654 > I1014 21:56:53.609491 56185 sched.cpp:365] Framework registered with > 201310142156-16777343-5432-56065-0000 > Registered > I1014 21:56:53.609565 56185 sched.cpp:379] Scheduler::registered took 15.017us > I1014 21:56:53.609279 56093 hierarchical_allocator_process.hpp:332] Added > framework 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.609715 56093 hierarchical_allocator_process.hpp:752] Offering > cpus(*):1; mem(*):96; disk(*):454895; ports(*):[31000-32000] on slave > 201310142156-16777343-5432-56065-0 to framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.610141 56093 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 468.125us > I1014 21:56:53.610266 56096 master.hpp:389] Adding offer > 201310142156-16777343-5432-56065-0 with resources cpus(*):1; mem(*):96; > disk(*):454895; ports(*):[31000-32000] on slave > 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) > I1014 21:56:53.610985 56096 master.cpp:1689] Sending 1 offers to framework > 201310142156-16777343-5432-56065-0000 > Resource offers received > Starting the task > I1014 21:56:53.611999 56196 sched.cpp:472] Scheduler::resourceOffers took > 99.793us > I1014 21:56:53.612448 56099 master.cpp:2015] Processing reply for offer > 201310142156-16777343-5432-56065-0 on slave > 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) for > framework 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.612907 56099 master.hpp:361] Adding task 1 with resources > mem(*):32 on slave 201310142156-16777343-5432-56065-0 > (smfd-atr-11-sr1.devel.twitter.com) > I1014 21:56:53.613489 56099 master.cpp:2139] Launching task 1 of framework > 201310142156-16777343-5432-56065-0000 with resources mem(*):32 on slave > 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) > I1014 21:56:53.614058 56092 hierarchical_allocator_process.hpp:547] Framework > 201310142156-16777343-5432-56065-0000 left cpus(*):1; disk(*):454895; > ports(*):[31000-32000] unused on slave 201310142156-16777343-5432-56065-0 > I1014 21:56:53.614356 56139 slave.cpp:786] Got assigned task 1 for framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.614683 56139 slave.cpp:897] Launching task 1 for framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.616145 56139 slave.cpp:1008] Queuing task '1' for executor > default of framework '201310142156-16777343-5432-56065-0000 > I1014 21:56:53.614069 56099 master.hpp:399] Removing offer > 201310142156-16777343-5432-56065-0 with resources cpus(*):1; mem(*):96; > disk(*):454895; ports(*):[31000-32000] on slave > 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) > I1014 21:56:53.616436 56142 slave.cpp:529] Successfully attached file > '/tmp/tmp.RrXIQ56113/slaves/201310142156-16777343-5432-56065-0/frameworks/201310142156-16777343-5432-56065-0000/executors/default/runs/1c847474-6b5e-47f9-acf7-a3f0a5f534ed' > I1014 21:56:53.614352 56092 hierarchical_allocator_process.hpp:590] Framework > 201310142156-16777343-5432-56065-0000 filtered slave > 201310142156-16777343-5432-56065-0 for 5secs > I1014 21:56:53.616778 56148 cgroups_isolator.cpp:516] Launching default > (/home/jyx/versions/mesos3/build/src/.libs/balloon-executor) in > /tmp/tmp.RrXIQ56113/slaves/201310142156-16777343-5432-56065-0/frameworks/201310142156-16777343-5432-56065-0000/executors/default/runs/1c847474-6b5e-47f9-acf7-a3f0a5f534ed > with resources mem(*):64 for framework 201310142156-16777343-5432-56065-0000 > in cgroup > mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed > I1014 21:56:53.617432 56148 cgroups_isolator.cpp:708] Changing cgroup > controls for executor default of framework > 201310142156-16777343-5432-56065-0000 with resources mem(*):64 > I1014 21:56:53.617730 56148 cgroups_isolator.cpp:1089] Updated > 'memory.soft_limit_in_bytes' to 64MB for executor default of framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.618276 56148 cgroups_isolator.cpp:1124] Updated > 'memory.limit_in_bytes' to 64MB for executor default of framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.619503 56148 cgroups_isolator.cpp:1177] Started listening for > OOM events for executor default of framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.620424 56148 cgroups_isolator.cpp:568] Forked executor at = > 56198 > I1014 21:56:53.649888 56149 slave.cpp:1460] Got registration for executor > 'default' of framework 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.650070 56132 cgroups_isolator.cpp:708] Changing cgroup > controls for executor default of framework > 201310142156-16777343-5432-56065-0000 with resources mem(*):96 > I1014 21:56:53.650190 56149 slave.cpp:1581] Flushing queued task 1 for > executor 'default' of framework 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.650724 56132 cgroups_isolator.cpp:1089] Updated > 'memory.soft_limit_in_bytes' to 96MB for executor default of framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.651286 56132 cgroups_isolator.cpp:1124] Updated > 'memory.limit_in_bytes' to 96MB for executor default of framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.651778 56132 cgroups_isolator.cpp:1200] Discarded memory > threshold (64MB) notifier for executor default of framework > 201310142156-16777343-5432-56065-0000 with uuid > 1c847474-6b5e-47f9-acf7-a3f0a5f534ed > I1014 21:56:53.757016 56148 cgroups_isolator.cpp:1226] OOM event detected for > executor default of framework 201310142156-16777343-5432-56065-0000 with uuid > 1c847474-6b5e-47f9-acf7-a3f0a5f534ed ; invoking OOM handler > I1014 21:56:53.757143 56148 cgroups_isolator.cpp:1272] OOM handler invoked > for executor default of framework 201310142156-16777343-5432-56065-0000 with > uuid 1c847474-6b5e-47f9-acf7-a3f0a5f534ed > I1014 21:56:53.976590 56148 cgroups_isolator.cpp:1311] Memory limit exceeded: > Requested: 96MB Maximum Used: 96MB > MEMORY STATISTICS: > cache 4096 > rss 100659200 > mapped_file 0 > pgpgin 24630 > pgpgout 54 > pgfault 27348 > pgmajfault 0 > inactive_anon 8200192 > active_anon 2347008 > inactive_file 4096 > active_file 0 > unevictable 90095616 > hierarchical_memory_limit 100663296 > total_cache 4096 > total_rss 100659200 > total_mapped_file 0 > total_pgpgin 24630 > total_pgpgout 54 > total_pgfault 27348 > total_pgmajfault 0 > total_inactive_anon 8257536 > total_active_anon 2347008 > total_inactive_file 4096 > total_active_file 0 > total_unevictable 90034176 > I1014 21:56:53.977507 56148 cgroups_isolator.cpp:672] Killing executor > default of framework 201310142156-16777343-5432-56065-0000 > I1014 21:56:53.979003 56148 cgroups_isolator.cpp:1200] Discarded memory > threshold (96MB) notifier for executor default of framework > 201310142156-16777343-5432-56065-0000 with uuid > 1c847474-6b5e-47f9-acf7-a3f0a5f534ed > I1014 21:56:53.997982 56153 process.cpp:986] Socket closed while receiving > I1014 21:56:54.084249 56130 cgroups.cpp:1193] Trying to freeze cgroup > /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed > I1014 21:56:54.085297 56130 cgroups.cpp:1232] Successfully froze cgroup > /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed > after 1 attempts > I1014 21:56:54.087805 56152 cgroups.cpp:1208] Trying to thaw cgroup > /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed > I1014 21:56:54.088615 56152 cgroups.cpp:1318] Successfully thawed > /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed > I1014 21:56:54.090497 56152 cgroups_isolator.cpp:1344] Successfully destroyed > cgroup > mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed > I1014 21:56:54.091753 56149 slave.cpp:2175] Executor 'default' of framework > 201310142156-16777343-5432-56065-0000 has terminated with signal Unknown > signal 127 > I1014 21:56:54.093579 56149 slave.cpp:1793] Handling status update > TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 from @0.0.0.0:0 > I1014 21:56:54.094251 56149 slave.cpp:3139] Terminating task 1 > I1014 21:56:54.095079 56088 master.cpp:1498] Executor default of framework > 201310142156-16777343-5432-56065-0000 on slave > 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) exited > with status -1 > I1014 21:56:54.094638 56148 cgroups_isolator.cpp:702] Asked to update > resources for an unknown/killed executor > I1014 21:56:54.094705 56149 status_update_manager.cpp:300] Received status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 > I1014 21:56:54.095487 56088 hierarchical_allocator_process.hpp:637] Recovered > mem(*):64 (total allocatable: cpus(*):1; disk(*):454895; > ports(*):[31000-32000]; mem(*):64) on slave > 201310142156-16777343-5432-56065-0 from framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:54.096093 56149 status_update_manager.cpp:471] Creating > StatusUpdate stream for task 1 of framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:54.096649 56149 status_update_manager.cpp:351] Forwarding status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432 > Task in state TASK_FAILED > Reason: Memory limit exceeded: Requested: 96MB Maximum Used: 96MB > MEMORY STATISTICS: > cache 4096 > rss 100659200 > mapped_file 0 > pgpgin 24630 > pgpgout 54 > pgfault 27348 > pgmajfault 0 > inactive_anon 8200192 > active_anon 2347008 > inactive_file 4096 > active_file 0 > unevictable 90095616 > hierarchical_memory_limit 100663296 > total_cache 4096 > total_rss 100659200 > total_mapped_file 0 > total_pgpgin 24630 > total_pgpgout 54 > total_pgfault 27348 > total_pgmajfault 0 > total_inactive_anon 8257536 > total_active_anon 2347008 > total_inactive_file 4096 > total_active_file 0 > total_unevictable 90034176 > I1014 21:56:54.097167 56149 slave.cpp:1912] Status update manager > successfully handled status update TASK_FAILED (UUID: > d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:54.098013 56192 sched.cpp:527] Scheduler::statusUpdate took > 57.234us > I1014 21:56:54.098096 56192 sched.cpp:654] Aborting framework > '201310142156-16777343-5432-56065-0000' > I1014 21:56:54.098150 56192 sched.cpp:542] Not sending status update > acknowledgment message because the driver is aborted! > I1014 21:56:54.097627 56091 master.cpp:1448] Status update TASK_FAILED (UUID: > d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework > 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 > I1014 21:56:54.098402 56091 master.hpp:379] Removing task 1 with resources > mem(*):32 on slave 201310142156-16777343-5432-56065-0 > (smfd-atr-11-sr1.devel.twitter.com) > I1014 21:56:54.098795 56091 master.cpp:1000] scheduler(1)@10.35.255.108:45654 > asked to deactivate framework 201310142156-16777343-5432-56065-0000 > I1014 21:56:54.098858 56086 hierarchical_allocator_process.hpp:637] Recovered > mem(*):32 (total allocatable: cpus(*):1; disk(*):454895; > ports(*):[31000-32000]; mem(*):96) on slave > 201310142156-16777343-5432-56065-0 from framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:54.098934 56091 master.cpp:1014] Deactivating framework > 201310142156-16777343-5432-56065-0000 > I1014 21:56:54.099546 56098 hierarchical_allocator_process.hpp:408] > Deactivated framework 201310142156-16777343-5432-56065-0000 > I1014 21:56:54.595873 56091 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:56:54.599139 56100 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 19.162us > I1014 21:56:55.600440 56102 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 20.653us > I1014 21:56:56.601618 56085 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 63.838us > I1014 21:56:57.602627 56096 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 24.951us > I1014 21:56:58.603863 56099 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 15.374us > I1014 21:56:58.604903 56182 sched.cpp:337] Ignoring authentication timeout > because the driver is aborted! > I1014 21:56:59.597434 56095 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:56:59.605849 56090 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 65.767us > I1014 21:57:00.606931 56092 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 22.189us > I1014 21:57:01.607993 56089 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 23.982us > I1014 21:57:02.609069 56080 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 27.794us > I1014 21:57:03.610221 56087 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 63.708us > W1014 21:57:04.098196 56141 status_update_manager.cpp:454] Resending status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 > I1014 21:57:04.099505 56141 status_update_manager.cpp:351] Forwarding status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432 > I1014 21:57:04.101228 56190 sched.cpp:502] Ignoring task status update > message because the driver is aborted! > W1014 21:57:04.100833 56088 master.cpp:1441] Status update TASK_FAILED (UUID: > d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework > 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 > (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task > I1014 21:57:04.598647 56086 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:57:04.611861 56081 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 13.172us > I1014 21:57:05.613046 56101 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 14.305us > I1014 21:57:06.614200 56098 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 25.934us > I1014 21:57:07.615419 56081 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 56.7us > I1014 21:57:08.616662 56097 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 83.339us > I1014 21:57:09.600827 56101 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:57:09.618006 56088 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 51.953us > I1014 21:57:10.619179 56086 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 63.038us > I1014 21:57:11.620412 56080 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 104.269us > I1014 21:57:12.621666 56100 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 18.998us > I1014 21:57:13.622977 56090 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 17.009us > W1014 21:57:14.101284 56149 status_update_manager.cpp:454] Resending status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 > I1014 21:57:14.102404 56149 status_update_manager.cpp:351] Forwarding status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432 > I1014 21:57:14.103987 56180 sched.cpp:502] Ignoring task status update > message because the driver is aborted! > W1014 21:57:14.103621 56089 master.cpp:1441] Status update TASK_FAILED (UUID: > d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework > 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 > (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task > I1014 21:57:14.602560 56085 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:57:14.624765 56099 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 20.175us > I1014 21:57:15.625892 56093 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 31.194us > I1014 21:57:16.626806 56082 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 22.441us > I1014 21:57:17.627913 56091 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 13.554us > I1014 21:57:18.629115 56101 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 13.692us > I1014 21:57:19.604230 56081 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:57:19.630450 56103 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 88.214us > I1014 21:57:20.631455 56094 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 14.402us > I1014 21:57:21.632642 56097 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 13.937us > I1014 21:57:22.634487 56085 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 66.372us > I1014 21:57:23.635640 56080 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 21.919us > W1014 21:57:24.103406 56139 status_update_manager.cpp:454] Resending status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 > I1014 21:57:24.104260 56139 status_update_manager.cpp:351] Forwarding status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432 > I1014 21:57:24.106067 56182 sched.cpp:502] Ignoring task status update > message because the driver is aborted! > W1014 21:57:24.105670 56103 master.cpp:1441] Status update TASK_FAILED (UUID: > d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework > 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 > (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task > I1014 21:57:24.605545 56082 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:57:24.636827 56091 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 15.822us > I1014 21:57:25.637984 56097 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 13.822us > I1014 21:57:26.639128 56094 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 15.062us > I1014 21:57:27.640239 56101 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 13.755us > I1014 21:57:28.641381 56089 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 18.619us > I1014 21:57:29.606472 56099 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:57:29.642806 56100 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 59.527us > I1014 21:57:30.644026 56083 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 21.656us > I1014 21:57:31.645145 56092 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 13.892us > I1014 21:57:32.646399 56082 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 12.527us > I1014 21:57:33.647538 56080 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 14.082us > W1014 21:57:34.105988 56140 status_update_manager.cpp:454] Resending status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 > I1014 21:57:34.106892 56140 status_update_manager.cpp:351] Forwarding status > update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of > framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432 > I1014 21:57:34.108701 56190 sched.cpp:502] Ignoring task status update > message because the driver is aborted! > W1014 21:57:34.108274 56102 master.cpp:1441] Status update TASK_FAILED (UUID: > d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework > 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 > (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task > I1014 21:57:34.608364 56084 master.cpp:85] No whitelist given. Advertising > offers for all slaves > I1014 21:57:34.648485 56085 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 14.287us > I1014 21:57:35.649591 56091 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 13.624us > I1014 21:57:36.650728 56098 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 14.247us > I1014 21:57:37.652891 56094 hierarchical_allocator_process.hpp:688] Performed > allocation for 1 slaves in 51.175us > ... -- This message was sent by Atlassian JIRA (v6.1#6144)