Hi Ben, I think the problem is mesos have found the executor on hdfs://master/user/mesos/hadoop.tar.gz, but it did not download it, so did not use it. Mesos found the executor, so it did not output error, just update the task status as lost; but mesos did not use the executor, so the executor directory contains nothing!
But I am not very familiar with source code, so I do not know why mesos can not use the executor. And I also do not know whether my analysis is right. Thanks very much for your help! Wang Yu 发件人： 王瑜 发送时间： 2013-05-15 11:04 收件人： mesos-dev 抄送： Benjamin Mahler 主题： 回复: 回复: org.apache.hadoop.mapred.MesosScheduler: Unknown/exited TaskTracker: http://slave5:50060 Hi, Ben, I have reworked the test, and checked log directory again, it is still null. The same as following. I think there is the problem with my executor, but I do not know how to let the executor works. Logs is as following... " Asked to update resources for an unknown/killed executor" why it always kill the executor? 1. I opened all the executor directory, but all of them are null. I do not know what happened to them... [root@slave1 logs]# cd /tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_4/runs/8a4dd631-1ec0-4946-a1bc-0644a7238e3c [root@slave1 8a4dd631-1ec0-4946-a1bc-0644a7238e3c]# ls [root@slave1 8a4dd631-1ec0-4946-a1bc-0644a7238e3c]# ls -l 总用量 0 [root@slave1 8a4dd631-1ec0-4946-a1bc-0644a7238e3c]# ls -a . .. [root@slave1 8a4dd631-1ec0-4946-a1bc-0644a7238e3c]# 2. I added "--isolation=cgroups" for slaves, but it still not work. Tasks are always lost. But there is no error any more, I still do not know what happened to the executor...Logs on one slave is as follows. Please help me, thanks very much! mesos-slave.INFO Log file created at: 2013/05/13 09:12:54 Running on machine: slave1 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0513 09:12:54.170383 24183 main.cpp:124] Creating "cgroups" isolator I0513 09:12:54.171617 24183 main.cpp:132] Build: 2013-04-10 16:07:43 by root I0513 09:12:54.171656 24183 main.cpp:133] Starting Mesos slave I0513 09:12:54.173495 24197 slave.cpp:203] Slave started on 1)@192.168.0.3:36668 I0513 09:12:54.173578 24197 slave.cpp:204] Slave resources: cpus=24; mem=63356; ports=[31000-32000]; disk=29143 I0513 09:12:54.174486 24192 cgroups_isolator.cpp:242] Using /cgroup as cgroups hierarchy root I0513 09:12:54.179914 24197 slave.cpp:453] New master detected at [email protected]:5050 I0513 09:12:54.180809 24197 slave.cpp:436] Successfully attached file '/home/mesos/build/logs/mesos-slave.INFO' I0513 09:12:54.180817 24207 status_update_manager.cpp:132] New master detected at [email protected]:5050 I0513 09:12:54.194345 24192 cgroups_isolator.cpp:730] Recovering isolator I0513 09:12:54.195453 24189 slave.cpp:377] Finished recovery I0513 09:12:54.197798 24206 slave.cpp:487] Registered with master; given slave ID 201305130913-33597632-5050-3893-0 I0513 09:12:54.198086 24201 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305081719-33597632-5050-4050-1' for removal I0513 09:12:54.198329 24201 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305100938-33597632-5050-19520-1' for removal I0513 09:12:54.198490 24201 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305081625-33597632-5050-2991-1' for removal I0513 09:12:54.198593 24201 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305081746-33597632-5050-12378-1' for removal I0513 09:12:54.198874 24201 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305090914-33597632-5050-5072-1' for removal I0513 09:12:54.199028 24201 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305081730-33597632-5050-8558-1' for removal I0513 09:12:54.199149 24201 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201304131144-33597632-5050-4949-2' for removal I0513 09:13:54.176460 24204 slave.cpp:1811] Current disk usage 26.93%. Max allowed age: 5.11days I0513 09:14:54.178444 24203 slave.cpp:1811] Current disk usage 26.93%. Max allowed age: 5.11days I0513 09:15:54.180680 24203 slave.cpp:1811] Current disk usage 26.93%. Max allowed age: 5.11days I0513 09:16:23.051203 24200 slave.cpp:587] Got assigned task Task_Tracker_0 for framework 201305130913-33597632-5050-3893-0000 I0513 09:16:23.054324 24200 paths.hpp:302] Created executor directory '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_0/runs/6522748a-9d43-41b7-8f88-cd537a502495' I0513 09:16:23.055605 24188 slave.cpp:436] Successfully attached file '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_0/runs/6522748a-9d43-41b7-8f88-cd537a502495' I0513 09:16:23.056043 24190 cgroups_isolator.cpp:525] Launching executor_Task_Tracker_0 (cd hadoop && ./bin/mesos-executor) in /tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_0/runs/6522748a-9d43-41b7-8f88-cd537a502495 with resources cpus=1; mem=1280 for framework 201305130913-33597632-5050-3893-0000 in cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_0_tag_6522748a-9d43-41b7-8f88-cd537a502495 I0513 09:16:23.059368 24190 cgroups_isolator.cpp:670] Changing cgroup controls for executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 with resources cpus=1; mem=1280 I0513 09:16:23.060478 24190 cgroups_isolator.cpp:841] Updated 'cpu.shares' to 1024 for executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:23.061101 24190 cgroups_isolator.cpp:979] Updated 'memory.limit_in_bytes' to 1342177280 for executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:23.061101 24190 cgroups_isolator.cpp:979] Updated 'memory.limit_in_bytes' to 1342177280 for executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:23.061807 24190 cgroups_isolator.cpp:1005] Started listening for OOM events for executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:23.063297 24190 cgroups_isolator.cpp:555] Forked executor at = 24552 I0513 09:16:29.055598 24190 slave.cpp:587] Got assigned task Task_Tracker_1 for framework 201305130913-33597632-5050-3893-0000 I0513 09:16:29.058297 24190 paths.hpp:302] Created executor directory '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_1/runs/38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b' I0513 09:16:29.059012 24203 slave.cpp:436] Successfully attached file '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_1/runs/38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b' I0513 09:16:29.059865 24200 cgroups_isolator.cpp:525] Launching executor_Task_Tracker_1 (cd hadoop && ./bin/mesos-executor) in /tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_1/runs/38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b with resources cpus=1; mem=1280 for framework 201305130913-33597632-5050-3893-0000 in cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_1_tag_38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b I0513 09:16:29.061282 24200 cgroups_isolator.cpp:670] Changing cgroup controls for executor executor_Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 with resources cpus=1; mem=1280 I0513 09:16:29.062208 24200 cgroups_isolator.cpp:841] Updated 'cpu.shares' to 1024 for executor executor_Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:29.062940 24200 cgroups_isolator.cpp:979] Updated 'memory.limit_in_bytes' to 1342177280 for executor executor_Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:29.063705 24200 cgroups_isolator.cpp:1005] Started listening for OOM events for executor executor_Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:29.065239 24200 cgroups_isolator.cpp:555] Forked executor at = 24628 I0513 09:16:34.457746 24188 cgroups_isolator.cpp:806] Executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 terminated with status 256 I0513 09:16:34.457909 24188 cgroups_isolator.cpp:635] Killing executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.459873 24188 cgroups_isolator.cpp:1025] OOM notifier is triggered for executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 with uuid 6522748a-9d43-41b7-8f88-cd537a502495 I0513 09:16:34.460028 24188 cgroups_isolator.cpp:1030] Discarded OOM notifier for executor executor_Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 with uuid 6522748a-9d43-41b7-8f88-cd537a502495 I0513 09:16:34.461314 24190 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_0_tag_6522748a-9d43-41b7-8f88-cd537a502495 I0513 09:16:34.461675 24190 cgroups.cpp:1214] Successfully froze cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_0_tag_6522748a-9d43-41b7-8f88-cd537a502495 after 1 attempts I0513 09:16:34.464400 24197 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_0_tag_6522748a-9d43-41b7-8f88-cd537a502495 I0513 09:16:34.464659 24197 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_0_tag_6522748a-9d43-41b7-8f88-cd537a502495 I0513 09:16:34.477118 24199 cgroups_isolator.cpp:1144] Successfully destroyed cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_0_tag_6522748a-9d43-41b7-8f88-cd537a502495 I0513 09:16:34.477439 24190 slave.cpp:1479] Executor 'executor_Task_Tracker_0' of framework 201305130913-33597632-5050-3893-0000 has exited with status 1 I0513 09:16:34.479852 24190 slave.cpp:1232] Handling status update TASK_LOST from task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.480123 24190 slave.cpp:1280] Forwarding status update TASK_LOST from task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 to the status update manager I0513 09:16:34.480136 24199 cgroups_isolator.cpp:666] Asked to update resources for an unknown/killed executor I0513 09:16:34.480480 24185 status_update_manager.cpp:254] Received status update TASK_LOST from task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.480716 24185 status_update_manager.cpp:403] Creating StatusUpdate stream for task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.480927 24185 status_update_manager.hpp:314] Handling UPDATE for status update TASK_LOST from task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.481107 24185 status_update_manager.cpp:289] Forwarding status update TASK_LOST from task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 to the master at [email protected]:5050 I0513 09:16:34.487007 24194 slave.cpp:979] Got acknowledgement of status update for task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.487257 24185 status_update_manager.cpp:314] Received status update acknowledgement for task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.487412 24185 status_update_manager.hpp:314] Handling ACK for status update TASK_LOST from task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.487547 24185 status_update_manager.cpp:434] Cleaning up status update stream for task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.487788 24207 slave.cpp:1016] Status update manager successfully handled status update acknowledgement for task Task_Tracker_0 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:34.488142 24202 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_0/runs/6522748a-9d43-41b7-8f88-cd537a502495' for removal I0513 09:16:35.063462 24199 slave.cpp:587] Got assigned task Task_Tracker_2 for framework 201305130913-33597632-5050-3893-0000 I0513 09:16:35.066090 24199 paths.hpp:302] Created executor directory '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_2/runs/f4729d73-5000-4c40-9c0e-1e77ad414f27' I0513 09:16:35.066673 24188 slave.cpp:436] Successfully attached file '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_2/runs/f4729d73-5000-4c40-9c0e-1e77ad414f27' I0513 09:16:35.066985 24205 cgroups_isolator.cpp:525] Launching executor_Task_Tracker_2 (cd hadoop && ./bin/mesos-executor) in /tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_2/runs/f4729d73-5000-4c40-9c0e-1e77ad414f27 with resources cpus=1; mem=1280 for framework 201305130913-33597632-5050-3893-0000 in cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_2_tag_f4729d73-5000-4c40-9c0e-1e77ad414f27 I0513 09:16:35.068594 24205 cgroups_isolator.cpp:670] Changing cgroup controls for executor executor_Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 with resources cpus=1; mem=1280 I0513 09:16:35.069341 24205 cgroups_isolator.cpp:841] Updated 'cpu.shares' to 1024 for executor executor_Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:35.070061 24205 cgroups_isolator.cpp:979] Updated 'memory.limit_in_bytes' to 1342177280 for executor executor_Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:35.070828 24205 cgroups_isolator.cpp:1005] Started listening for OOM events for executor executor_Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:35.071966 24205 cgroups_isolator.cpp:555] Forked executor at = 24704 I0513 09:16:40.464987 24197 cgroups_isolator.cpp:806] Executor executor_Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 terminated with status 256 I0513 09:16:40.465175 24197 cgroups_isolator.cpp:635] Killing executor executor_Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.467118 24197 cgroups_isolator.cpp:1025] OOM notifier is triggered for executor executor_Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 with uuid 38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b I0513 09:16:40.467269 24197 cgroups_isolator.cpp:1030] Discarded OOM notifier for executor executor_Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 with uuid 38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b I0513 09:16:40.468596 24198 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_1_tag_38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b I0513 09:16:40.468945 24198 cgroups.cpp:1214] Successfully froze cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_1_tag_38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b after 1 attempts I0513 09:16:40.471577 24200 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_1_tag_38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b I0513 09:16:40.471850 24200 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_1_tag_38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b I0513 09:16:40.480960 24185 cgroups_isolator.cpp:1144] Successfully destroyed cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_1_tag_38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b I0513 09:16:40.481230 24196 slave.cpp:1479] Executor 'executor_Task_Tracker_1' of framework 201305130913-33597632-5050-3893-0000 has exited with status 1 I0513 09:16:40.483572 24196 slave.cpp:1232] Handling status update TASK_LOST from task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.483801 24196 slave.cpp:1280] Forwarding status update TASK_LOST from task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 to the status update manager I0513 09:16:40.483846 24193 cgroups_isolator.cpp:666] Asked to update resources for an unknown/killed executor I0513 09:16:40.484094 24205 status_update_manager.cpp:254] Received status update TASK_LOST from task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.484267 24205 status_update_manager.cpp:403] Creating StatusUpdate stream for task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.484412 24205 status_update_manager.hpp:314] Handling UPDATE for status update TASK_LOST from task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.484558 24205 status_update_manager.cpp:289] Forwarding status update TASK_LOST from task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 to the master at [email protected]:5050 I0513 09:16:40.487229 24202 slave.cpp:979] Got acknowledgement of status update for task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.487457 24196 status_update_manager.cpp:314] Received status update acknowledgement for task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.487607 24196 status_update_manager.hpp:314] Handling ACK for status update TASK_LOST from task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.487741 24196 status_update_manager.cpp:434] Cleaning up status update stream for task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.487949 24207 slave.cpp:1016] Status update manager successfully handled status update acknowledgement for task Task_Tracker_1 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:40.488278 24193 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_1/runs/38d83d3a-ef9d-4118-b28c-c6c3cfba6c4b' for removal I0513 09:16:41.072098 24194 slave.cpp:587] Got assigned task Task_Tracker_3 for framework 201305130913-33597632-5050-3893-0000 I0513 09:16:41.074632 24194 paths.hpp:302] Created executor directory '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_3/runs/22f6e84b-d07f-430a-a322-6f804b3cd642' I0513 09:16:41.075546 24198 slave.cpp:436] Successfully attached file '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_3/runs/22f6e84b-d07f-430a-a322-6f804b3cd642' I0513 09:16:41.076081 24194 cgroups_isolator.cpp:525] Launching executor_Task_Tracker_3 (cd hadoop && ./bin/mesos-executor) in /tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_3/runs/22f6e84b-d07f-430a-a322-6f804b3cd642 with resources cpus=1; mem=1280 for framework 201305130913-33597632-5050-3893-0000 in cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_3_tag_22f6e84b-d07f-430a-a322-6f804b3cd642 I0513 09:16:41.077606 24194 cgroups_isolator.cpp:670] Changing cgroup controls for executor executor_Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 with resources cpus=1; mem=1280 I0513 09:16:41.078402 24194 cgroups_isolator.cpp:841] Updated 'cpu.shares' to 1024 for executor executor_Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:41.079186 24194 cgroups_isolator.cpp:979] Updated 'memory.limit_in_bytes' to 1342177280 for executor executor_Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:41.080008 24194 cgroups_isolator.cpp:1005] Started listening for OOM events for executor executor_Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:41.081447 24194 cgroups_isolator.cpp:555] Forked executor at = 24780 I0513 09:16:44.482589 24200 status_update_manager.cpp:379] Checking for unacknowledged status updates I0513 09:16:46.473145 24199 cgroups_isolator.cpp:806] Executor executor_Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 terminated with status 256 I0513 09:16:46.473307 24199 cgroups_isolator.cpp:635] Killing executor executor_Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.475491 24199 cgroups_isolator.cpp:1025] OOM notifier is triggered for executor executor_Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 with uuid f4729d73-5000-4c40-9c0e-1e77ad414f27 I0513 09:16:46.475649 24199 cgroups_isolator.cpp:1030] Discarded OOM notifier for executor executor_Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 with uuid f4729d73-5000-4c40-9c0e-1e77ad414f27 I0513 09:16:46.476820 24192 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_2_tag_f4729d73-5000-4c40-9c0e-1e77ad414f27 I0513 09:16:46.477181 24192 cgroups.cpp:1214] Successfully froze cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_2_tag_f4729d73-5000-4c40-9c0e-1e77ad414f27 after 1 attempts I0513 09:16:46.479907 24201 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_2_tag_f4729d73-5000-4c40-9c0e-1e77ad414f27 I0513 09:16:46.480229 24201 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_2_tag_f4729d73-5000-4c40-9c0e-1e77ad414f27 I0513 09:16:46.493069 24200 cgroups_isolator.cpp:1144] Successfully destroyed cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_2_tag_f4729d73-5000-4c40-9c0e-1e77ad414f27 I0513 09:16:46.493391 24184 slave.cpp:1479] Executor 'executor_Task_Tracker_2' of framework 201305130913-33597632-5050-3893-0000 has exited with status 1 I0513 09:16:46.495689 24184 slave.cpp:1232] Handling status update TASK_LOST from task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.495933 24184 slave.cpp:1280] Forwarding status update TASK_LOST from task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 to the status update manager I0513 09:16:46.495980 24189 cgroups_isolator.cpp:666] Asked to update resources for an unknown/killed executor I0513 09:16:46.496305 24193 status_update_manager.cpp:254] Received status update TASK_LOST from task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.496553 24193 status_update_manager.cpp:403] Creating StatusUpdate stream for task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.496707 24193 status_update_manager.hpp:314] Handling UPDATE for status update TASK_LOST from task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.496868 24193 status_update_manager.cpp:289] Forwarding status update TASK_LOST from task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 to the master at [email protected]:5050 I0513 09:16:46.499631 24201 slave.cpp:979] Got acknowledgement of status update for task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.499961 24193 status_update_manager.cpp:314] Received status update acknowledgement for task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.500128 24193 status_update_manager.hpp:314] Handling ACK for status update TASK_LOST from task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.500257 24193 status_update_manager.cpp:434] Cleaning up status update stream for task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.500452 24192 slave.cpp:1016] Status update manager successfully handled status update acknowledgement for task Task_Tracker_2 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:46.500743 24204 gc.cpp:56] Scheduling '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_2/runs/f4729d73-5000-4c40-9c0e-1e77ad414f27' for removal I0513 09:16:47.079013 24193 slave.cpp:587] Got assigned task Task_Tracker_4 for framework 201305130913-33597632-5050-3893-0000 I0513 09:16:47.081650 24193 paths.hpp:302] Created executor directory '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_4/runs/8a4dd631-1ec0-4946-a1bc-0644a7238e3c' I0513 09:16:47.082447 24198 slave.cpp:436] Successfully attached file '/tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_4/runs/8a4dd631-1ec0-4946-a1bc-0644a7238e3c' I0513 09:16:47.082861 24194 cgroups_isolator.cpp:525] Launching executor_Task_Tracker_4 (cd hadoop && ./bin/mesos-executor) in /tmp/mesos/slaves/201305130913-33597632-5050-3893-0/frameworks/201305130913-33597632-5050-3893-0000/executors/executor_Task_Tracker_4/runs/8a4dd631-1ec0-4946-a1bc-0644a7238e3c with resources cpus=1; mem=1280 for framework 201305130913-33597632-5050-3893-0000 in cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_4_tag_8a4dd631-1ec0-4946-a1bc-0644a7238e3c I0513 09:16:47.084478 24194 cgroups_isolator.cpp:670] Changing cgroup controls for executor executor_Task_Tracker_4 of framework 201305130913-33597632-5050-3893-0000 with resources cpus=1; mem=1280 I0513 09:16:47.085273 24194 cgroups_isolator.cpp:841] Updated 'cpu.shares' to 1024 for executor executor_Task_Tracker_4 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:47.086045 24194 cgroups_isolator.cpp:979] Updated 'memory.limit_in_bytes' to 1342177280 for executor executor_Task_Tracker_4 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:47.086853 24194 cgroups_isolator.cpp:1005] Started listening for OOM events for executor executor_Task_Tracker_4 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:47.088227 24194 cgroups_isolator.cpp:555] Forked executor at = 24856 I0513 09:16:50.485791 24194 status_update_manager.cpp:379] Checking for unacknowledged status updates I0513 09:16:52.480471 24185 cgroups_isolator.cpp:806] Executor executor_Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 terminated with status 256 I0513 09:16:52.480622 24185 cgroups_isolator.cpp:635] Killing executor executor_Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 I0513 09:16:52.482652 24185 cgroups_isolator.cpp:1025] OOM notifier is triggered for executor executor_Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 with uuid 22f6e84b-d07f-430a-a322-6f804b3cd642 I0513 09:16:52.482805 24185 cgroups_isolator.cpp:1030] Discarded OOM notifier for executor executor_Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 with uuid 22f6e84b-d07f-430a-a322-6f804b3cd642 I0513 09:16:52.484110 24195 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_3_tag_22f6e84b-d07f-430a-a322-6f804b3cd642 I0513 09:16:52.484447 24195 cgroups.cpp:1214] Successfully froze cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_3_tag_22f6e84b-d07f-430a-a322-6f804b3cd642 after 1 attempts I0513 09:16:52.487893 24184 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_3_tag_22f6e84b-d07f-430a-a322-6f804b3cd642 I0513 09:16:52.488129 24184 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_3_tag_22f6e84b-d07f-430a-a322-6f804b3cd642 I0513 09:16:52.496047 24207 cgroups_isolator.cpp:1144] Successfully destroyed cgroup mesos/framework_201305130913-33597632-5050-3893-0000_executor_executor_Task_Tracker_3_tag_22f6e84b-d07f-430a-a322-6f804b3cd642 I0513 09:16:52.496247 24203 slave.cpp:1479] Executor 'executor_Task_Tracker_3' of framework 201305130913-33597632-5050-3893-0000 has exited with status 1 I0513 09:16:52.498538 24203 slave.cpp:1232] Handling status update TASK_LOST from task Task_Tracker_3 of framework 201305130913-33597632-5050-3893-0000 ...... Wang Yu
