[
https://issues.apache.org/jira/browse/MESOS-738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kone resolved MESOS-738.
------------------------------
Resolution: Fixed
commit 30f8d4764249ab22beb5f13966ec46ed528d24b4
> CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework_NoBuffer can't finish
> -----------------------------------------------------------------------
>
> Key: MESOS-738
> URL: https://issues.apache.org/jira/browse/MESOS-738
> Project: Mesos
> Issue Type: Bug
> Reporter: Yan Xu
> Assignee: Vinod Kone
> Fix For: 0.15.0
>
>
> This doesn't seem to be a problem on Jenkins. Wondering what was causing the
> problem on my test box...
> Test command:
> sudo MESOS_VERBOSE=1 GLOG_v=1 make check -j
> GTEST_FILTER="CgroupsIsolatorTest/ROOT_CGROUPS_BalloonFramework*"
> Log:
> [ RUN ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework_NoBuffer
> Using temporary directory
> '/tmp/CgroupsIsolatorTest_ROOT_CGROUPS_BalloonFramework_NoBuffer_UkYn2q'
> Launched master at 56065
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1014 21:56:49.593399 56065 process.cpp:1555] libprocess is initialized on
> 127.0.0.1:5432 for 24 cpus
> I1014 21:56:49.593868 56065 logging.cpp:110] Logging to STDERR
> I1014 21:56:49.594128 56065 main.cpp:114] Build: 2013-10-14 18:18:08 by root
> I1014 21:56:49.594156 56065 main.cpp:115] Starting Mesos master
> I1014 21:56:49.594457 56084 master.cpp:284] Master started on 127.0.0.1:5432
> I1014 21:56:49.594609 56084 master.cpp:299] Master ID:
> 201310142156-16777343-5432-56065
> I1014 21:56:49.594671 56084 master.cpp:302] Master only allowing
> authenticated frameworks to register!
> I1014 21:56:49.595432 56101 hierarchical_allocator_process.hpp:302]
> Initializing hierarchical allocator process with master :
> [email protected]:5432
> I1014 21:56:49.595450 56090 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:56:49.596568 56084 master.cpp:697] Elected as master!
> I1014 21:56:50.595949 56086 hierarchical_allocator_process.hpp:726] No
> resources available to allocate!
> I1014 21:56:50.596043 56086 hierarchical_allocator_process.hpp:688] Performed
> allocation for 0 slaves in 108.976us
> Launched slave at 56114
> I1014 21:56:51.596371 56089 hierarchical_allocator_process.hpp:726] No
> resources available to allocate!
> I1014 21:56:51.596426 56089 hierarchical_allocator_process.hpp:688] Performed
> allocation for 0 slaves in 66.034us
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1014 21:56:51.600865 56114 process.cpp:1555] libprocess is initialized on
> 10.35.255.108:5051 for 24 cpus
> I1014 21:56:51.601160 56114 logging.cpp:110] Logging to STDERR
> I1014 21:56:51.601503 56114 main.cpp:119] Creating "cgroups" isolator
> I1014 21:56:51.601678 56114 main.cpp:127] Build: 2013-10-14 18:18:08 by root
> I1014 21:56:51.601706 56114 main.cpp:128] Starting Mesos slave
> I1014 21:56:51.603461 56137 slave.cpp:108] Slave started on
> 1)@10.35.255.108:5051
> I1014 21:56:51.604014 56137 slave.cpp:208] Slave resources: cpus(*):1;
> mem(*):96; disk(*):454895; ports(*):[31000-32000]
> I1014 21:56:51.605553 56138 cgroups_isolator.cpp:224] Using
> /tmp/mesos_test_cgroup as cgroups hierarchy root
> I1014 21:56:51.606158 56137 slave.cpp:547] New master detected at
> [email protected]:5432
> I1014 21:56:51.606353 56139 status_update_manager.cpp:157] New master
> detected at [email protected]:5432
> I1014 21:56:51.606314 56137 slave.cpp:562] Postponing registration until
> recovery is complete
> I1014 21:56:52.597699 56086 hierarchical_allocator_process.hpp:726] No
> resources available to allocate!
> I1014 21:56:52.597754 56086 hierarchical_allocator_process.hpp:688] Performed
> allocation for 0 slaves in 64.23us
> I1014 21:56:52.604190 56138 cgroups_isolator.cpp:817] Recovering isolator
> I1014 21:56:52.604665 56138 slave.cpp:399] Finished recovery
> I1014 21:56:52.605499 56089 master.cpp:1248] Attempting to register slave on
> smfd-atr-11-sr1.devel.twitter.com at slave(1)@10.35.255.108:5051
> I1014 21:56:52.605533 56089 master.cpp:2502] Adding slave
> 201310142156-16777343-5432-56065-0 at smfd-atr-11-sr1.devel.twitter.com with
> cpus(*):1; mem(*):96; disk(*):454895; ports(*):[31000-32000]
> I1014 21:56:52.605934 56080 hierarchical_allocator_process.hpp:445] Added
> slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
> with cpus(*):1; mem(*):96; disk(*):454895; ports(*):[31000-32000] (and
> cpus(*):1; mem(*):96; disk(*):454895; ports(*):[31000-32000] available)
> I1014 21:56:52.606008 56080 hierarchical_allocator_process.hpp:708] Performed
> allocation for slave 201310142156-16777343-5432-56065-0 in 17.439us
> I1014 21:56:52.605998 56146 slave.cpp:613] Registered with master
> [email protected]:5432; given slave ID 201310142156-16777343-5432-56065-0
> Enabling authentication for the framework
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1014 21:56:53.598536 56084 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 25.985us
> I1014 21:56:53.598472 56159 process.cpp:1555] libprocess is initialized on
> 10.35.255.108:45654 for 24 cpus
> I1014 21:56:53.599010 56159 logging.cpp:110] Logging to STDERR
> I1014 21:56:53.601263 56182 sched.cpp:195] New master at [email protected]:5432
> I1014 21:56:53.601438 56182 sched.cpp:281] Authenticating with master
> [email protected]:5432
> I1014 21:56:53.601858 56186 authenticatee.hpp:100] Initializing client SASL
> I1014 21:56:53.603885 56186 authenticatee.hpp:124] Creating new client SASL
> connection
> I1014 21:56:53.604517 56089 master.cpp:1723] Authenticating framework at
> scheduler(1)@10.35.255.108:45654
> I1014 21:56:53.604909 56102 authenticator.hpp:83] Initializing server SASL
> I1014 21:56:53.606436 56102 auxprop.cpp:45] Initialized in-memory auxiliary
> property plugin
> I1014 21:56:53.606469 56102 authenticator.hpp:140] Creating new server SASL
> connection
> I1014 21:56:53.606798 56182 authenticatee.hpp:212] Received SASL
> authentication mechanisms: CRAM-MD5
> I1014 21:56:53.606838 56182 authenticatee.hpp:238] Attempting to authenticate
> with mechanism 'CRAM-MD5'
> I1014 21:56:53.607024 56102 authenticator.hpp:243] Received SASL
> authentication start
> I1014 21:56:53.607133 56102 authenticator.hpp:325] Authentication requires
> more steps
> I1014 21:56:53.607283 56191 authenticatee.hpp:258] Received SASL
> authentication step
> I1014 21:56:53.607574 56096 authenticator.hpp:271] Received SASL
> authentication step
> I1014 21:56:53.607641 56096 auxprop.cpp:81] Request to lookup properties for
> user: 'test-principal' realm: 'smfd-atr-11-sr1.devel.twitter.com' server
> FQDN: 'smfd-atr-11-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false
> SASL_AUXPROP_AUTHZID: false
> I1014 21:56:53.607661 56096 auxprop.cpp:153] Looking up auxiliary property
> '*userPassword'
> I1014 21:56:53.607683 56096 auxprop.cpp:153] Looking up auxiliary property
> '*cmusaslsecretCRAM-MD5'
> I1014 21:56:53.607699 56096 auxprop.cpp:81] Request to lookup properties for
> user: 'test-principal' realm: 'smfd-atr-11-sr1.devel.twitter.com' server
> FQDN: 'smfd-atr-11-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false
> SASL_AUXPROP_AUTHZID: true
> I1014 21:56:53.607712 56096 auxprop.cpp:103] Skipping auxiliary property
> '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I1014 21:56:53.607724 56096 auxprop.cpp:103] Skipping auxiliary property
> '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I1014 21:56:53.607745 56096 authenticator.hpp:317] Authentication success
> I1014 21:56:53.608036 56192 authenticatee.hpp:298] Authentication success
> I1014 21:56:53.608242 56196 sched.cpp:326] Successfully authenticated with
> master [email protected]:5432
> I1014 21:56:53.607980 56093 master.cpp:1763] Successfully authenticated
> framework at scheduler(1)@10.35.255.108:45654
> I1014 21:56:53.608603 56093 master.cpp:768] Received registration request
> from scheduler(1)@10.35.255.108:45654
> I1014 21:56:53.608747 56093 master.cpp:786] Registering framework
> 201310142156-16777343-5432-56065-0000 at scheduler(1)@10.35.255.108:45654
> I1014 21:56:53.609491 56185 sched.cpp:365] Framework registered with
> 201310142156-16777343-5432-56065-0000
> Registered
> I1014 21:56:53.609565 56185 sched.cpp:379] Scheduler::registered took 15.017us
> I1014 21:56:53.609279 56093 hierarchical_allocator_process.hpp:332] Added
> framework 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.609715 56093 hierarchical_allocator_process.hpp:752] Offering
> cpus(*):1; mem(*):96; disk(*):454895; ports(*):[31000-32000] on slave
> 201310142156-16777343-5432-56065-0 to framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.610141 56093 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 468.125us
> I1014 21:56:53.610266 56096 master.hpp:389] Adding offer
> 201310142156-16777343-5432-56065-0 with resources cpus(*):1; mem(*):96;
> disk(*):454895; ports(*):[31000-32000] on slave
> 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
> I1014 21:56:53.610985 56096 master.cpp:1689] Sending 1 offers to framework
> 201310142156-16777343-5432-56065-0000
> Resource offers received
> Starting the task
> I1014 21:56:53.611999 56196 sched.cpp:472] Scheduler::resourceOffers took
> 99.793us
> I1014 21:56:53.612448 56099 master.cpp:2015] Processing reply for offer
> 201310142156-16777343-5432-56065-0 on slave
> 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) for
> framework 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.612907 56099 master.hpp:361] Adding task 1 with resources
> mem(*):32 on slave 201310142156-16777343-5432-56065-0
> (smfd-atr-11-sr1.devel.twitter.com)
> I1014 21:56:53.613489 56099 master.cpp:2139] Launching task 1 of framework
> 201310142156-16777343-5432-56065-0000 with resources mem(*):32 on slave
> 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
> I1014 21:56:53.614058 56092 hierarchical_allocator_process.hpp:547] Framework
> 201310142156-16777343-5432-56065-0000 left cpus(*):1; disk(*):454895;
> ports(*):[31000-32000] unused on slave 201310142156-16777343-5432-56065-0
> I1014 21:56:53.614356 56139 slave.cpp:786] Got assigned task 1 for framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.614683 56139 slave.cpp:897] Launching task 1 for framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.616145 56139 slave.cpp:1008] Queuing task '1' for executor
> default of framework '201310142156-16777343-5432-56065-0000
> I1014 21:56:53.614069 56099 master.hpp:399] Removing offer
> 201310142156-16777343-5432-56065-0 with resources cpus(*):1; mem(*):96;
> disk(*):454895; ports(*):[31000-32000] on slave
> 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
> I1014 21:56:53.616436 56142 slave.cpp:529] Successfully attached file
> '/tmp/tmp.RrXIQ56113/slaves/201310142156-16777343-5432-56065-0/frameworks/201310142156-16777343-5432-56065-0000/executors/default/runs/1c847474-6b5e-47f9-acf7-a3f0a5f534ed'
> I1014 21:56:53.614352 56092 hierarchical_allocator_process.hpp:590] Framework
> 201310142156-16777343-5432-56065-0000 filtered slave
> 201310142156-16777343-5432-56065-0 for 5secs
> I1014 21:56:53.616778 56148 cgroups_isolator.cpp:516] Launching default
> (/home/jyx/versions/mesos3/build/src/.libs/balloon-executor) in
> /tmp/tmp.RrXIQ56113/slaves/201310142156-16777343-5432-56065-0/frameworks/201310142156-16777343-5432-56065-0000/executors/default/runs/1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> with resources mem(*):64 for framework 201310142156-16777343-5432-56065-0000
> in cgroup
> mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> I1014 21:56:53.617432 56148 cgroups_isolator.cpp:708] Changing cgroup
> controls for executor default of framework
> 201310142156-16777343-5432-56065-0000 with resources mem(*):64
> I1014 21:56:53.617730 56148 cgroups_isolator.cpp:1089] Updated
> 'memory.soft_limit_in_bytes' to 64MB for executor default of framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.618276 56148 cgroups_isolator.cpp:1124] Updated
> 'memory.limit_in_bytes' to 64MB for executor default of framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.619503 56148 cgroups_isolator.cpp:1177] Started listening for
> OOM events for executor default of framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.620424 56148 cgroups_isolator.cpp:568] Forked executor at =
> 56198
> I1014 21:56:53.649888 56149 slave.cpp:1460] Got registration for executor
> 'default' of framework 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.650070 56132 cgroups_isolator.cpp:708] Changing cgroup
> controls for executor default of framework
> 201310142156-16777343-5432-56065-0000 with resources mem(*):96
> I1014 21:56:53.650190 56149 slave.cpp:1581] Flushing queued task 1 for
> executor 'default' of framework 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.650724 56132 cgroups_isolator.cpp:1089] Updated
> 'memory.soft_limit_in_bytes' to 96MB for executor default of framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.651286 56132 cgroups_isolator.cpp:1124] Updated
> 'memory.limit_in_bytes' to 96MB for executor default of framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.651778 56132 cgroups_isolator.cpp:1200] Discarded memory
> threshold (64MB) notifier for executor default of framework
> 201310142156-16777343-5432-56065-0000 with uuid
> 1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> I1014 21:56:53.757016 56148 cgroups_isolator.cpp:1226] OOM event detected for
> executor default of framework 201310142156-16777343-5432-56065-0000 with uuid
> 1c847474-6b5e-47f9-acf7-a3f0a5f534ed ; invoking OOM handler
> I1014 21:56:53.757143 56148 cgroups_isolator.cpp:1272] OOM handler invoked
> for executor default of framework 201310142156-16777343-5432-56065-0000 with
> uuid 1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> I1014 21:56:53.976590 56148 cgroups_isolator.cpp:1311] Memory limit exceeded:
> Requested: 96MB Maximum Used: 96MB
> MEMORY STATISTICS:
> cache 4096
> rss 100659200
> mapped_file 0
> pgpgin 24630
> pgpgout 54
> pgfault 27348
> pgmajfault 0
> inactive_anon 8200192
> active_anon 2347008
> inactive_file 4096
> active_file 0
> unevictable 90095616
> hierarchical_memory_limit 100663296
> total_cache 4096
> total_rss 100659200
> total_mapped_file 0
> total_pgpgin 24630
> total_pgpgout 54
> total_pgfault 27348
> total_pgmajfault 0
> total_inactive_anon 8257536
> total_active_anon 2347008
> total_inactive_file 4096
> total_active_file 0
> total_unevictable 90034176
> I1014 21:56:53.977507 56148 cgroups_isolator.cpp:672] Killing executor
> default of framework 201310142156-16777343-5432-56065-0000
> I1014 21:56:53.979003 56148 cgroups_isolator.cpp:1200] Discarded memory
> threshold (96MB) notifier for executor default of framework
> 201310142156-16777343-5432-56065-0000 with uuid
> 1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> I1014 21:56:53.997982 56153 process.cpp:986] Socket closed while receiving
> I1014 21:56:54.084249 56130 cgroups.cpp:1193] Trying to freeze cgroup
> /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> I1014 21:56:54.085297 56130 cgroups.cpp:1232] Successfully froze cgroup
> /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> after 1 attempts
> I1014 21:56:54.087805 56152 cgroups.cpp:1208] Trying to thaw cgroup
> /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> I1014 21:56:54.088615 56152 cgroups.cpp:1318] Successfully thawed
> /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> I1014 21:56:54.090497 56152 cgroups_isolator.cpp:1344] Successfully destroyed
> cgroup
> mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
> I1014 21:56:54.091753 56149 slave.cpp:2175] Executor 'default' of framework
> 201310142156-16777343-5432-56065-0000 has terminated with signal Unknown
> signal 127
> I1014 21:56:54.093579 56149 slave.cpp:1793] Handling status update
> TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000 from @0.0.0.0:0
> I1014 21:56:54.094251 56149 slave.cpp:3139] Terminating task 1
> I1014 21:56:54.095079 56088 master.cpp:1498] Executor default of framework
> 201310142156-16777343-5432-56065-0000 on slave
> 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) exited
> with status -1
> I1014 21:56:54.094638 56148 cgroups_isolator.cpp:702] Asked to update
> resources for an unknown/killed executor
> I1014 21:56:54.094705 56149 status_update_manager.cpp:300] Received status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000
> I1014 21:56:54.095487 56088 hierarchical_allocator_process.hpp:637] Recovered
> mem(*):64 (total allocatable: cpus(*):1; disk(*):454895;
> ports(*):[31000-32000]; mem(*):64) on slave
> 201310142156-16777343-5432-56065-0 from framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:54.096093 56149 status_update_manager.cpp:471] Creating
> StatusUpdate stream for task 1 of framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:54.096649 56149 status_update_manager.cpp:351] Forwarding status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000 to [email protected]:5432
> Task in state TASK_FAILED
> Reason: Memory limit exceeded: Requested: 96MB Maximum Used: 96MB
> MEMORY STATISTICS:
> cache 4096
> rss 100659200
> mapped_file 0
> pgpgin 24630
> pgpgout 54
> pgfault 27348
> pgmajfault 0
> inactive_anon 8200192
> active_anon 2347008
> inactive_file 4096
> active_file 0
> unevictable 90095616
> hierarchical_memory_limit 100663296
> total_cache 4096
> total_rss 100659200
> total_mapped_file 0
> total_pgpgin 24630
> total_pgpgout 54
> total_pgfault 27348
> total_pgmajfault 0
> total_inactive_anon 8257536
> total_active_anon 2347008
> total_inactive_file 4096
> total_active_file 0
> total_unevictable 90034176
> I1014 21:56:54.097167 56149 slave.cpp:1912] Status update manager
> successfully handled status update TASK_FAILED (UUID:
> d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:54.098013 56192 sched.cpp:527] Scheduler::statusUpdate took
> 57.234us
> I1014 21:56:54.098096 56192 sched.cpp:654] Aborting framework
> '201310142156-16777343-5432-56065-0000'
> I1014 21:56:54.098150 56192 sched.cpp:542] Not sending status update
> acknowledgment message because the driver is aborted!
> I1014 21:56:54.097627 56091 master.cpp:1448] Status update TASK_FAILED (UUID:
> d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework
> 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051
> I1014 21:56:54.098402 56091 master.hpp:379] Removing task 1 with resources
> mem(*):32 on slave 201310142156-16777343-5432-56065-0
> (smfd-atr-11-sr1.devel.twitter.com)
> I1014 21:56:54.098795 56091 master.cpp:1000] scheduler(1)@10.35.255.108:45654
> asked to deactivate framework 201310142156-16777343-5432-56065-0000
> I1014 21:56:54.098858 56086 hierarchical_allocator_process.hpp:637] Recovered
> mem(*):32 (total allocatable: cpus(*):1; disk(*):454895;
> ports(*):[31000-32000]; mem(*):96) on slave
> 201310142156-16777343-5432-56065-0 from framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:54.098934 56091 master.cpp:1014] Deactivating framework
> 201310142156-16777343-5432-56065-0000
> I1014 21:56:54.099546 56098 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201310142156-16777343-5432-56065-0000
> I1014 21:56:54.595873 56091 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:56:54.599139 56100 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 19.162us
> I1014 21:56:55.600440 56102 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 20.653us
> I1014 21:56:56.601618 56085 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 63.838us
> I1014 21:56:57.602627 56096 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 24.951us
> I1014 21:56:58.603863 56099 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 15.374us
> I1014 21:56:58.604903 56182 sched.cpp:337] Ignoring authentication timeout
> because the driver is aborted!
> I1014 21:56:59.597434 56095 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:56:59.605849 56090 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 65.767us
> I1014 21:57:00.606931 56092 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 22.189us
> I1014 21:57:01.607993 56089 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 23.982us
> I1014 21:57:02.609069 56080 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 27.794us
> I1014 21:57:03.610221 56087 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 63.708us
> W1014 21:57:04.098196 56141 status_update_manager.cpp:454] Resending status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000
> I1014 21:57:04.099505 56141 status_update_manager.cpp:351] Forwarding status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000 to [email protected]:5432
> I1014 21:57:04.101228 56190 sched.cpp:502] Ignoring task status update
> message because the driver is aborted!
> W1014 21:57:04.100833 56088 master.cpp:1441] Status update TASK_FAILED (UUID:
> d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework
> 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051
> (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task
> I1014 21:57:04.598647 56086 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:57:04.611861 56081 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 13.172us
> I1014 21:57:05.613046 56101 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 14.305us
> I1014 21:57:06.614200 56098 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 25.934us
> I1014 21:57:07.615419 56081 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 56.7us
> I1014 21:57:08.616662 56097 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 83.339us
> I1014 21:57:09.600827 56101 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:57:09.618006 56088 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 51.953us
> I1014 21:57:10.619179 56086 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 63.038us
> I1014 21:57:11.620412 56080 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 104.269us
> I1014 21:57:12.621666 56100 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 18.998us
> I1014 21:57:13.622977 56090 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 17.009us
> W1014 21:57:14.101284 56149 status_update_manager.cpp:454] Resending status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000
> I1014 21:57:14.102404 56149 status_update_manager.cpp:351] Forwarding status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000 to [email protected]:5432
> I1014 21:57:14.103987 56180 sched.cpp:502] Ignoring task status update
> message because the driver is aborted!
> W1014 21:57:14.103621 56089 master.cpp:1441] Status update TASK_FAILED (UUID:
> d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework
> 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051
> (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task
> I1014 21:57:14.602560 56085 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:57:14.624765 56099 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 20.175us
> I1014 21:57:15.625892 56093 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 31.194us
> I1014 21:57:16.626806 56082 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 22.441us
> I1014 21:57:17.627913 56091 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 13.554us
> I1014 21:57:18.629115 56101 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 13.692us
> I1014 21:57:19.604230 56081 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:57:19.630450 56103 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 88.214us
> I1014 21:57:20.631455 56094 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 14.402us
> I1014 21:57:21.632642 56097 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 13.937us
> I1014 21:57:22.634487 56085 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 66.372us
> I1014 21:57:23.635640 56080 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 21.919us
> W1014 21:57:24.103406 56139 status_update_manager.cpp:454] Resending status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000
> I1014 21:57:24.104260 56139 status_update_manager.cpp:351] Forwarding status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000 to [email protected]:5432
> I1014 21:57:24.106067 56182 sched.cpp:502] Ignoring task status update
> message because the driver is aborted!
> W1014 21:57:24.105670 56103 master.cpp:1441] Status update TASK_FAILED (UUID:
> d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework
> 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051
> (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task
> I1014 21:57:24.605545 56082 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:57:24.636827 56091 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 15.822us
> I1014 21:57:25.637984 56097 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 13.822us
> I1014 21:57:26.639128 56094 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 15.062us
> I1014 21:57:27.640239 56101 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 13.755us
> I1014 21:57:28.641381 56089 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 18.619us
> I1014 21:57:29.606472 56099 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:57:29.642806 56100 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 59.527us
> I1014 21:57:30.644026 56083 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 21.656us
> I1014 21:57:31.645145 56092 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 13.892us
> I1014 21:57:32.646399 56082 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 12.527us
> I1014 21:57:33.647538 56080 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 14.082us
> W1014 21:57:34.105988 56140 status_update_manager.cpp:454] Resending status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000
> I1014 21:57:34.106892 56140 status_update_manager.cpp:351] Forwarding status
> update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of
> framework 201310142156-16777343-5432-56065-0000 to [email protected]:5432
> I1014 21:57:34.108701 56190 sched.cpp:502] Ignoring task status update
> message because the driver is aborted!
> W1014 21:57:34.108274 56102 master.cpp:1441] Status update TASK_FAILED (UUID:
> d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework
> 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051
> (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task
> I1014 21:57:34.608364 56084 master.cpp:85] No whitelist given. Advertising
> offers for all slaves
> I1014 21:57:34.648485 56085 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 14.287us
> I1014 21:57:35.649591 56091 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 13.624us
> I1014 21:57:36.650728 56098 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 14.247us
> I1014 21:57:37.652891 56094 hierarchical_allocator_process.hpp:688] Performed
> allocation for 1 slaves in 51.175us
> ...
--
This message was sent by Atlassian JIRA
(v6.1#6144)
