when the executor is unhealthy, the slave sends a kill command to the executor. only when the executor terminates, does the slave transitions all its tasks to LOST and inform the master and hence the scheduler. from your observation, sounds like the kill signal was either being trapped by the executor or was not being properly sent to the executor. is the task/executor Task_Tracker_242 still running on the slave? you can check this with 'ps'. what does the executor logs say. did it receive any signal?
On Tue, Jul 9, 2013 at 10:37 PM, 王国栋 <wangg...@gmail.com> wrote: > No log about killing Task_Tracker_242. But I can see Task_Tracker_242 is in > STAGING from master/slave web UI. And it is stuck. The hadoop framework > consider that Task_Tracker_242 is launched, though it is not running. > > I think if slave deems Task_Tracker_242 unhealthy, it should report to > master that this task is Lost. But I am not sure why it can not report > this. I am trying to go through the code. > > Guodong > > > On Wed, Jul 10, 2013 at 2:51 AM, Vinod Kone <vinodk...@gmail.com> wrote: > > > Hey Guodong, > > > > So, looks like Task_Tracker_242 did not register with the slave within 1 > > minute and the slave decided to kill it because it was deemed unhealthy. > At > > this point the executor should've received a kill signal from the slave. > Do > > you see anything of that sort in the slave or executor logs? > > > > > > On Mon, Jul 8, 2013 at 11:30 PM, 王国栋 <wangg...@gmail.com> wrote: > > > > > Hi vinod. > > > > > > I am using the code from the trunk. I think the latest commit is at Jul > > > 1st. I will grep some master log in another mail. > > > > > > The Task "Task_Tracker_242" is stuck in STAGING. I think > > "Task_Tracker_224" > > > and "Task_Tracker_230" exit sucessfully. But it is strange that there > > are a > > > lot of "Fail to collect resource..." warnings. > > > > > > I0709 00:46:11.288698 11002 slave.cpp:739] Got assigned task > > > Task_Tracker_242 for framework 201307040929-252063498-5050-27411-0000 > > > I0709 00:46:11.289136 11002 slave.cpp:837] Launching task > > Task_Tracker_242 > > > for framework 201307040929-252063498-5050-27411-0000 > > > I0709 00:46:11.291296 11002 paths.hpp:303] Created executor directory > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > cutor_Task_Tracker_242/runs/5c47ad99-1c78-43c8-9f27-9509f1d39c3d' > > > I0709 00:46:11.291647 11002 slave.cpp:948] Queuing task > > 'Task_Tracker_242' > > > for executor executor_Task_Tracker_242 of framework > > > '201307040929-252063498-5050-27411-0000 > > > I0709 00:46:11.292162 11002 slave.cpp:511] Successfully attached file > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > cutor_Task_Tracker_242/runs/5c47ad99-1c78-43c8-9f27-9509f1d39c3d' > > > W0709 00:46:12.197242 10992 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:16.100548 10994 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:17.197463 11001 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:21.101570 11002 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:22.198303 11005 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:26.102522 11002 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:27.199403 10998 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:31.103610 10998 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:32.200248 11001 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:36.104547 11004 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:37.201236 10991 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:41.105523 10997 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:42.202250 10991 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > I0709 00:46:45.283098 11002 slave.cpp:2511] Current usage 57.43%. Max > > > allowed age: 2.279812884766227days > > > W0709 00:46:46.106760 10994 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:47.203474 10993 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:51.107544 11006 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:52.204280 10997 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:56.108530 10995 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:46:57.205417 10997 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:47:01.109284 10997 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:47:02.206368 11002 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > I0709 00:47:05.288517 11002 slave.cpp:2463] Terminating executor > > > executor_Task_Tracker_238 of framework > > > 201307040929-252063498-5050-27411-0000 because it did not register > within > > > 1mins > > > W0709 00:47:06.110532 11005 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:47:07.207320 10997 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_230' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > W0709 00:47:11.111778 10996 monitor.cpp:186] Failed to collect resource > > > usage for executor 'executor_Task_Tracker_224' of framework > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > I0709 00:47:11.292485 10991 slave.cpp:2463] Terminating executor > > > executor_Task_Tracker_242 of framework > > > 201307040929-252063498-5050-27411-0000 because it did not register > within > > > 1mins > > > > > > > > > Guodong > > > > > > > > > On Tue, Jul 9, 2013 at 2:21 PM, Vinod Kone <vinodk...@gmail.com> > wrote: > > > > > > > hey guodong, which of these task(s) is stuck in STAGING? also, the > > > > corresponding master's logs would also be helpful here. also which > > > version > > > > of mesos are you running? > > > > > > > > > > > > On Mon, Jul 8, 2013 at 11:02 PM, 王国栋 <wangg...@gmail.com> wrote: > > > > > > > > > It is very interesting that there are these logs. > > > > > > > > > > I0709 00:33:43.833853 11002 slave.cpp:996] Asked to kill task > > > > > Task_Tracker_224 of framework > 201307040929-252063498-5050-27411-0000 > > > > > I0709 00:33:43.835552 11006 slave.cpp:996] Asked to kill task > > > > > Task_Tracker_230 of framework > 201307040929-252063498-5050-27411-0000 > > > > > I0709 00:33:43.972771 10994 slave.cpp:1692] Handling status update > > > > > TASK_FINISHED (UUID: 372081cc-edf2-4183-a461-9345ab6d279c) for task > > > > > Task_Tracker_230 of framework > 201307040929-252063498-5050-27411-0000 > > > > > from executor(1)@10.47.6.21:27786 > > > > > I0709 00:33:43.973132 10994 status_update_manager.cpp:290] Received > > > > status > > > > > update TASK_FINISHED (UUID: 372081cc-edf2-4183-a461-9345ab6d279c) > for > > > > task > > > > > Task_Tracker_230 of framework 201307040929-252063498-5 > > > > > 050-27411-0000 with checkpoint=false > > > > > I0709 00:33:43.973192 10994 status_update_manager.cpp:336] > Forwarding > > > > > status update TASK_FINISHED (UUID: > > > 372081cc-edf2-4183-a461-9345ab6d279c) > > > > > for task Task_Tracker_230 of framework 201307040929-252063498 > > > > > -5050-27411-0000 to master@10.47.6.15:5050 > > > > > I0709 00:33:43.973846 11005 slave.cpp:1809] Sending acknowledgement > > for > > > > > status update TASK_FINISHED (UUID: > > > 372081cc-edf2-4183-a461-9345ab6d279c) > > > > > for task Task_Tracker_230 of framework 201307040929-2520634 > > > > > 98-5050-27411-0000 to executor(1)@10.47.6.21:27786 > > > > > I0709 00:33:43.974591 11000 status_update_manager.cpp:360] Received > > > > status > > > > > update acknowledgement 372081cc-edf2-4183-a461-9345ab6d279c for > task > > > > > Task_Tracker_230 of framework 201307040929-252063498-5050-27 > > > > > 411-0000 > > > > > I0709 00:33:43.974652 11000 status_update_manager.cpp:481] Cleaning > > up > > > > > status update stream for task Task_Tracker_230 of framework > > > > > 201307040929-252063498-5050-27411-0000 > > > > > I0709 00:33:44.090603 11003 slave.cpp:1692] Handling status update > > > > > TASK_FINISHED (UUID: 61d5775a-2375-412a-a5a4-80ab55163d88) for task > > > > > Task_Tracker_224 of framework > 201307040929-252063498-5050-27411-0000 > > > > > from executor(1)@10.47.6.21:2310 > > > > > I0709 00:33:44.090860 11003 status_update_manager.cpp:290] Received > > > > status > > > > > update TASK_FINISHED (UUID: 61d5775a-2375-412a-a5a4-80ab55163d88) > for > > > > task > > > > > Task_Tracker_224 of framework 201307040929-252063498-5 > > > > > 050-27411-0000 with checkpoint=false > > > > > I0709 00:33:44.090973 11003 status_update_manager.cpp:336] > Forwarding > > > > > status update TASK_FINISHED (UUID: > > > 61d5775a-2375-412a-a5a4-80ab55163d88) > > > > > for task Task_Tracker_224 of framework 201307040929-252063498 > > > > > -5050-27411-0000 to master@10.47.6.15:5050 > > > > > I0709 00:33:44.091279 11003 slave.cpp:1809] Sending acknowledgement > > for > > > > > status update TASK_FINISHED (UUID: > > > 61d5775a-2375-412a-a5a4-80ab55163d88) > > > > > for task Task_Tracker_224 of framework 201307040929-2520634 > > > > > 98-5050-27411-0000 to executor(1)@10.47.6.21:2310 > > > > > I0709 00:33:44.093286 11003 status_update_manager.cpp:360] Received > > > > status > > > > > update acknowledgement 61d5775a-2375-412a-a5a4-80ab55163d88 for > task > > > > > Task_Tracker_224 of framework 201307040929-252063498-5050-27 > > > > > 411-0000 > > > > > I0709 00:33:44.093359 11003 status_update_manager.cpp:481] Cleaning > > up > > > > > status update stream for task Task_Tracker_224 of framework > > > > > 201307040929-252063498-5050-27411-0000 > > > > > I0709 00:33:45.259831 10997 slave.cpp:2511] Current usage 57.44%. > Max > > > > > allowed age: 2.279168852469954days > > > > > W0709 00:33:45.949470 10996 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:33:47.063181 11005 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:33:50.950412 11000 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:33:52.063576 10993 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:33:55.951427 11003 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:33:57.064575 10998 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:34:00.952390 11003 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:34:02.065218 10998 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:34:05.953456 10995 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:34:07.066515 10995 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:34:10.954479 10998 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:34:12.067471 11005 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:34:15.955461 10996 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > W0709 00:34:17.068209 10996 monitor.cpp:186] Failed to collect > > resource > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > > > > > > > > > > > > > > Guodong > > > > > > > > > > > > > > > On Tue, Jul 9, 2013 at 1:59 PM, 王国栋 <wangg...@gmail.com> wrote: > > > > > > > > > > > Hi ben, > > > > > > > > > > > > I ran into the same issue here. > > > > > > > > > > > > This also happens in our hadoop framework. The slave log is like > > > these. > > > > > At > > > > > > that time, I think the work load of the node is very high. > > > > > > > > > > > > I0708 23:36:44.253880 11005 slave.cpp:739] Got assigned task > > > > > > Task_Tracker_224 for framework > > 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:44.255221 10999 gc.cpp:84] Unscheduling > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000' > > > > > > for removal > > > > > > I0708 23:36:44.256206 11001 slave.cpp:837] Launching task > > > > > Task_Tracker_224 > > > > > > for framework 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:44.258117 11001 paths.hpp:303] Created executor > > directory > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > > > cutor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1' > > > > > > I0708 23:36:44.258467 10991 process_isolator.cpp:99] Launching > > > > > > executor_Task_Tracker_224 (cd hadoop && ./bin/mesos-executor) in > > > > > > > > > > > > > > > > > > > > > /data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/framew > > > > > > > > > > > > > > > > > > > > > orks/201307040929-252063498-5050-27411-0000/executors/executor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1 > > > > > > with resources cpus=1; mem=1280' for framework > > > > > > 201307040929-252063498-5050-27411-0 > > > > > > 000 > > > > > > I0708 23:36:44.258496 11001 slave.cpp:948] Queuing task > > > > > 'Task_Tracker_224' > > > > > > for executor executor_Task_Tracker_224 of framework > > > > > > '201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:44.261446 10991 process_isolator.cpp:161] Forked > > executor > > > > at > > > > > > 2220 > > > > > > I0708 23:36:44.261787 10996 slave.cpp:511] Successfully attached > > file > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > > > cutor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1' > > > > > > I0708 23:36:44.580497 10996 slave.cpp:2511] Current usage 57.21%. > > Max > > > > > > allowed age: 2.295155852123924days > > > > > > I0708 23:36:44.750393 11002 slave.cpp:1395] Got registration for > > > > executor > > > > > > 'executor_Task_Tracker_224' of framework > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:44.751095 11002 slave.cpp:1510] Flushing queued task > > > > > > Task_Tracker_224 for executor 'executor_Task_Tracker_224' of > > > framework > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:46.144317 11006 slave.cpp:1692] Handling status > update > > > > > > TASK_RUNNING (UUID: 364ee347-f6a2-4c7b-8702-460aa0ece579) for > task > > > > > > Task_Tracker_224 of framework > > 201307040929-252063498-5050-27411-0000 > > > f > > > > > > rom executor(1)@10.47.6.21:2310 > > > > > > I0708 23:36:46.144745 11006 status_update_manager.cpp:290] > Received > > > > > status > > > > > > update TASK_RUNNING (UUID: 364ee347-f6a2-4c7b-8702-460aa0ece579) > > for > > > > task > > > > > > Task_Tracker_224 of framework 201307040929-252063498-50 > > > > > > 50-27411-0000 with checkpoint=false > > > > > > I0708 23:36:46.144821 11006 status_update_manager.cpp:450] > Creating > > > > > > StatusUpdate stream for task Task_Tracker_224 of framework > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:46.145076 11006 status_update_manager.cpp:336] > > Forwarding > > > > > > status update TASK_RUNNING (UUID: > > > 364ee347-f6a2-4c7b-8702-460aa0ece579) > > > > > for > > > > > > task Task_Tracker_224 of framework 201307040929-252063498- > > > > > > 5050-27411-0000 to master@10.47.6.15:5050 > > > > > > I0708 23:36:46.145882 10997 slave.cpp:1809] Sending > acknowledgement > > > for > > > > > > status update TASK_RUNNING (UUID: > > > 364ee347-f6a2-4c7b-8702-460aa0ece579) > > > > > for > > > > > > task Task_Tracker_224 of framework 201307040929-25206349 > > > > > > 8-5050-27411-0000 to executor(1)@10.47.6.21:2310 > > > > > > I0708 23:36:46.146870 10993 status_update_manager.cpp:360] > Received > > > > > status > > > > > > update acknowledgement 364ee347-f6a2-4c7b-8702-460aa0ece579 for > > task > > > > > > Task_Tracker_224 of framework 201307040929-252063498-5050-27 > > > > > > 411-0000 > > > > > > I0708 23:36:50.258347 11005 slave.cpp:739] Got assigned task > > > > > > Task_Tracker_230 for framework > > 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:50.259472 11005 slave.cpp:837] Launching task > > > > > Task_Tracker_230 > > > > > > for framework 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:50.261641 11005 paths.hpp:303] Created executor > > directory > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > > > cutor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd' > > > > > > I0708 23:36:50.262265 11005 slave.cpp:948] Queuing task > > > > > 'Task_Tracker_230' > > > > > > for executor executor_Task_Tracker_230 of framework > > > > > > '201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:50.262557 11005 process_isolator.cpp:99] Launching > > > > > > executor_Task_Tracker_230 (cd hadoop && ./bin/mesos-executor) in > > > > > > > > > > > > > > > > > > > > > /data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/framew > > > > > > > > > > > > > > > > > > > > > orks/201307040929-252063498-5050-27411-0000/executors/executor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd > > > > > > with resources cpus=1; mem=1280' for framework > > > > > > 201307040929-252063498-5050-27411-0 > > > > > > 000 > > > > > > I0708 23:36:50.265396 10999 slave.cpp:511] Successfully attached > > file > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > > > cutor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd' > > > > > > I0708 23:36:50.265419 11005 process_isolator.cpp:161] Forked > > executor > > > > at > > > > > > 2851 > > > > > > I0708 23:36:50.835607 10995 slave.cpp:1395] Got registration for > > > > executor > > > > > > 'executor_Task_Tracker_230' of framework > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:50.836174 10995 slave.cpp:1510] Flushing queued task > > > > > > Task_Tracker_230 for executor 'executor_Task_Tracker_230' of > > > framework > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:54.617856 10994 slave.cpp:1692] Handling status > update > > > > > > TASK_RUNNING (UUID: 7753252d-c90b-4b0d-adca-7c97f38f692e) for > task > > > > > > Task_Tracker_230 of framework > > 201307040929-252063498-5050-27411-0000 > > > f > > > > > > rom executor(1)@10.47.6.21:27786 > > > > > > I0708 23:36:54.618275 10994 status_update_manager.cpp:290] > Received > > > > > status > > > > > > update TASK_RUNNING (UUID: 7753252d-c90b-4b0d-adca-7c97f38f692e) > > for > > > > task > > > > > > Task_Tracker_230 of framework 201307040929-252063498-50 > > > > > > 50-27411-0000 with checkpoint=false > > > > > > I0708 23:36:54.618326 10994 status_update_manager.cpp:450] > Creating > > > > > > StatusUpdate stream for task Task_Tracker_230 of framework > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > I0708 23:36:54.618443 10994 status_update_manager.cpp:336] > > Forwarding > > > > > > status update TASK_RUNNING (UUID: > > > 7753252d-c90b-4b0d-adca-7c97f38f692e) > > > > > for > > > > > > task Task_Tracker_230 of framework 201307040929-252063498- > > > > > > 5050-27411-0000 to master@10.47.6.15:5050 > > > > > > I0708 23:36:54.619137 10994 slave.cpp:1809] Sending > acknowledgement > > > for > > > > > > status update TASK_RUNNING (UUID: > > > 7753252d-c90b-4b0d-adca-7c97f38f692e) > > > > > for > > > > > > task Task_Tracker_230 of framework 201307040929-25206349 > > > > > > 8-5050-27411-0000 to executor(1)@10.47.6.21:27786 > > > > > > I0708 23:36:54.637682 10994 status_update_manager.cpp:360] > Received > > > > > status > > > > > > update acknowledgement 7753252d-c90b-4b0d-adca-7c97f38f692e for > > task > > > > > > Task_Tracker_230 of framework 201307040929-252063498-5050-27 > > > > > > 411-0000 > > > > > > I0708 23:37:44.583014 11002 slave.cpp:2511] Current usage 57.23%. > > Max > > > > > > allowed age: 2.293704423241597days > > > > > > I0708 23:38:44.585233 11003 slave.cpp:2511] Current usage 57.23%. > > Max > > > > > > allowed age: 2.293703916528542days > > > > > > I0708 23:39:44.599442 11006 slave.cpp:2511] Current usage 57.23%. > > Max > > > > > > allowed age: 2.293639867998055days > > > > > > I0708 23:40:44.603996 10997 slave.cpp:2511] Current usage 57.24%. > > Max > > > > > > allowed age: 2.292921551567535days > > > > > > I0708 23:41:44.608608 11006 slave.cpp:2511] Current usage 57.26%. > > Max > > > > > > allowed age: 2.291521098018820days > > > > > > I0708 23:42:44.609956 10992 slave.cpp:2511] Current usage 57.23%. > > Max > > > > > > allowed age: 2.293668041244063days > > > > > > I0708 23:43:44.682621 11000 slave.cpp:2511] Current usage 57.24%. > > Max > > > > > > allowed age: 2.292935638190544days > > > > > > I0708 23:44:44.684306 10993 slave.cpp:2511] Current usage 57.24%. > > Max > > > > > > allowed age: 2.292916079066516days > > > > > > I0708 23:45:44.686172 11001 slave.cpp:2511] Current usage 57.26%. > > Max > > > > > > allowed age: 2.291485324076945days > > > > > > I0708 23:46:44.699095 10995 slave.cpp:2511] Current usage 57.23%. > > Max > > > > > > allowed age: 2.293641894850289days > > > > > > I0708 23:47:44.721156 10998 slave.cpp:2511] Current usage 57.23%. > > Max > > > > > > allowed age: 2.293629429709074days > > > > > > I0708 23:48:44.779767 10992 slave.cpp:2511] Current usage 57.24%. > > Max > > > > > > allowed age: 2.293525350847025days > > > > > > I0708 23:49:44.812389 11004 slave.cpp:2511] Current usage 57.24%. > > Max > > > > > > allowed age: 2.292909289111539days > > > > > > I0708 23:50:44.814146 10999 slave.cpp:2511] Current usage 57.27%. > > Max > > > > > > allowed age: 2.291438098419977days > > > > > > I0708 23:51:44.814877 11005 slave.cpp:2511] Current usage 57.23%. > > Max > > > > > > allowed age: 2.293635104895313days > > > > > > I0708 23:52:44.818620 10998 slave.cpp:2511] Current usage 57.24%. > > Max > > > > > > allowed age: 2.292983775931019days > > > > > > I0708 23:53:44.829911 10997 slave.cpp:2511] Current usage 57.33%. > > Max > > > > > > allowed age: 2.286910009194236days > > > > > > I0708 23:54:44.831307 10999 slave.cpp:2511] Current usage 57.33%. > > Max > > > > > > allowed age: 2.286909502481169days > > > > > > I0708 23:55:44.902858 10994 slave.cpp:2511] Current usage 57.37%. > > Max > > > > > > allowed age: 2.284414244700093days > > > > > > I0708 23:56:44.905398 11002 slave.cpp:2511] Current usage 57.42%. > > Max > > > > > > allowed age: 2.280636901540567days > > > > > > I0708 23:57:44.933673 10991 slave.cpp:2511] Current usage 57.44%. > > Max > > > > > > allowed age: 2.279481899796968days > > > > > > I0708 23:58:44.934840 11004 slave.cpp:2511] Current usage 57.48%. > > Max > > > > > > allowed age: 2.276566475548496days > > > > > > I0708 23:59:44.936063 11001 slave.cpp:2511] Current usage 57.49%. > > Max > > > > > > allowed age: 2.275690368671817days > > > > > > I0709 00:00:44.937433 11004 slave.cpp:2511] Current usage 57.50%. > > Max > > > > > > allowed age: 2.275057180034989days > > > > > > I0709 00:01:44.938940 11001 slave.cpp:2511] Current usage 57.51%. > > Max > > > > > > allowed age: 2.273999467198449days > > > > > > I0709 00:02:44.955103 10996 slave.cpp:2511] Current usage 57.52%. > > Max > > > > > > allowed age: 2.273472384275891days > > > > > > I0709 00:03:44.956354 10993 slave.cpp:2511] Current usage 57.39%. > > Max > > > > > > allowed age: 2.282894612240220days > > > > > > I0709 00:04:44.957926 10997 slave.cpp:2511] Current usage 57.40%. > > Max > > > > > > allowed age: 2.281966516603831days > > > > > > I0709 00:05:44.969205 10996 slave.cpp:2511] Current usage 57.40%. > > Max > > > > > > allowed age: 2.281962260214144days > > > > > > I0709 00:06:44.969987 10992 slave.cpp:2511] Current usage 57.40%. > > Max > > > > > > allowed age: 2.281791801941551days > > > > > > I0709 00:07:44.977504 11004 slave.cpp:2511] Current usage 57.40%. > > Max > > > > > > allowed age: 2.281715288269849days > > > > > > I0709 00:08:44.982868 10998 slave.cpp:2511] Current usage 57.40%. > > Max > > > > > > allowed age: 2.281699782850289days > > > > > > I0709 00:09:44.997082 11000 slave.cpp:2511] Current usage 57.42%. > > Max > > > > > > allowed age: 2.280776044946192days > > > > > > I0709 00:10:44.998754 10994 slave.cpp:2511] Current usage 57.42%. > > Max > > > > > > allowed age: 2.280772193926956days > > > > > > I0709 00:11:44.999949 11002 slave.cpp:2511] Current usage 57.44%. > > Max > > > > > > allowed age: 2.279204525069213days > > > > > > I0709 00:12:45.001539 10995 slave.cpp:2511] Current usage 57.47%. > > Max > > > > > > allowed age: 2.277132676719109days > > > > > > I0709 00:13:45.002728 10992 slave.cpp:2511] Current usage 57.43%. > > Max > > > > > > allowed age: 2.280012428368322days > > > > > > I0709 00:14:45.009699 10998 slave.cpp:2511] Current usage 57.48%. > > Max > > > > > > allowed age: 2.276733690857512days > > > > > > I0709 00:15:45.013483 10996 slave.cpp:2511] Current usage 57.53%. > > Max > > > > > > allowed age: 2.272715152282546days > > > > > > I0709 00:16:45.015496 10998 slave.cpp:2511] Current usage 57.57%. > > Max > > > > > > allowed age: 2.270354274804352days > > > > > > I0709 00:17:45.016628 11000 slave.cpp:2511] Current usage 57.62%. > > Max > > > > > > allowed age: 2.266927678423322days > > > > > > I0709 00:18:45.032670 11002 slave.cpp:2511] Current usage 57.65%. > > Max > > > > > > allowed age: 2.264218182361482days > > > > > > I0709 00:19:45.043442 10998 slave.cpp:2511] Current usage 57.69%. > > Max > > > > > > allowed age: 2.261509598383137days > > > > > > I0709 00:20:45.080648 10992 slave.cpp:2511] Current usage 57.72%. > > Max > > > > > > allowed age: 2.259379478031400days > > > > > > I0709 00:21:45.081632 10995 slave.cpp:2511] Current usage 57.77%. > > Max > > > > > > allowed age: 2.255819920144039days > > > > > > I0709 00:22:45.082593 11005 slave.cpp:2511] Current usage 57.81%. > > Max > > > > > > allowed age: 2.253314528101817days > > > > > > I0709 00:23:45.193588 10997 slave.cpp:2511] Current usage 57.85%. > > Max > > > > > > allowed age: 2.250524870034248days > > > > > > I0709 00:24:45.220617 10994 slave.cpp:2511] Current usage 57.90%. > > Max > > > > > > allowed age: 2.246784618270532days > > > > > > I0709 00:25:45.241602 10992 slave.cpp:2511] Current usage 57.97%. > > Max > > > > > > allowed age: 2.242399422127049days > > > > > > I0709 00:26:45.248977 11000 slave.cpp:2511] Current usage 58.00%. > > Max > > > > > > allowed age: 2.240250654734792days > > > > > > I0709 00:27:45.250953 10993 slave.cpp:2511] Current usage 57.99%. > > Max > > > > > > allowed age: 2.240516983117894days > > > > > > I0709 00:28:45.252694 10996 slave.cpp:2511] Current usage 58.06%. > > Max > > > > > > allowed age: 2.235834143724352days > > > > > > I0709 00:29:45.254992 11003 slave.cpp:2511] Current usage 58.10%. > > Max > > > > > > allowed age: 2.233297436815162days > > > > > > W0709 00:30:06.753098 10999 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:10.715373 10996 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:11.754446 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:15.719880 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:16.755473 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:20.720330 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:21.766019 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:25.721364 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:26.768874 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:30.722605 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:31.770354 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:35.724455 10992 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:36.788751 10992 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:40.745380 10992 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:30:41.789358 10992 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > I0709 00:30:45.256590 11004 slave.cpp:2511] Current usage 58.11%. > > Max > > > > > > allowed age: 2.232469873049410days > > > > > > > > > > > > > > > > > > Guodong > > > > > > > > > > > > > > > > > > On Tue, Jul 9, 2013 at 4:55 AM, Benjamin Mahler < > > > > > benjamin.mah...@gmail.com > > > > > > > wrote: > > > > > > > > > > > >> Are these the un-edited logs? I'm expecting to see some logs > from > > > the > > > > > >> process_isolator or cgroups_isolator in there. > > > > > >> > > > > > >> > > > > > >> On Fri, Jul 5, 2013 at 2:38 PM, Brenden Matthews < > > > > > >> brenden.matth...@airbedandbreakfast.com> wrote: > > > > > >> > > > > > >> > Hey guys, > > > > > >> > > > > > > >> > I'm currently having a problem where tasks will get stuck in > the > > > > > staging > > > > > >> > state, though according to the logs they should have been > > > > terminated. > > > > > >> They > > > > > >> > hang indefinitely, or until I restart the slave. Below is a > > > > > screenshot > > > > > >> + > > > > > >> > logs. Also interesting is the 'Failed to collect resource > usage > > > > ...' > > > > > >> > messages. > > > > > >> > > > > > > >> > [image: Inline image 2] > > > > > >> > > > > > > >> > I0705 16:19:51.551512 9706 slave.cpp:739] Got assigned task > > > > > >> >> > > ct:1373041190990:0:add_latest_reservation_survey_events_partitio > > > > > >> >> n for framework chronos > > > > > >> >> I0705 16:19:51.552150 9706 slave.cpp:837] Launching task > > > > > >> >> > > > ct:1373041190990:0:add_latest_reservation_survey_events_partition f > > > > > >> >> or framework chronos > > > > > >> >> I0705 16:19:51.553956 9706 paths.hpp:303] Created executor > > > > directory > > > > > >> >> '/tmp/mesos/slaves/201307030043-2037266954-5050-15277-1 > > > > > >> >> > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > 517/frameworks/chronos/executors/ct:1373041190990:0:add_latest_reservation_survey_events_partition/runs/611ba128-557f-4b5e-8c > > > > > >> >> f2-4d1ce60d618f' > > > > > >> >> I0705 16:19:51.554576 9706 slave.cpp:948] Queuing task > > > > > >> >> > > > > 'ct:1373041190990:0:add_latest_reservation_survey_events_partition' f > > > > > >> >> or executor > > > > > >> >> > > ct:1373041190990:0:add_latest_reservation_survey_events_partition > > > > of > > > > > >> >> framework 'c > > > > > >> >> hronos > > > > > >> >> I0705 16:19:51.555027 9706 slave.cpp:511] Successfully > > attached > > > > file > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > '/tmp/mesos/slaves/201307030043-2037266954-5050-15277-1517/frameworks/chronos/executors/ct:1373041190990:0:add_latest_reservation_survey_events_partition/runs/611ba128-557f-4b5e-8cf2-4d1ce60d618f' > > > > > >> >> I0705 16:19:54.048754 9724 slave.cpp:2530] Current usage > > 42.18%. > > > > Max > > > > > >> >> allowed age: 22.955009563956388hrs > > > > > >> >> W0705 16:19:54.108963 9724 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:19:59.110787 9729 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:04.112406 9704 monitor.cpp:186] Failed to > collect > > > > > >> resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:09.114367 9705 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:14.116312 9706 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:19.118370 9699 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:24.120311 9701 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:29.122355 9700 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:34.123443 9722 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:39.125660 9718 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:44.127464 9724 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:49.129385 9725 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> I0705 16:20:51.555174 9703 slave.cpp:2482] Terminating > > executor > > > > > >> >> > > ct:1373041190990:0:add_latest_reservation_survey_events_partition > > > > of > > > > > >> >> framework chronos because it did not register within 1mins > > > > > >> >> I0705 16:20:54.050434 9717 slave.cpp:2530] Current usage > > 42.18%. > > > > Max > > > > > >> >> allowed age: 22.955009342481944hrs > > > > > >> >> W0705 16:20:54.130730 9699 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:20:59.132472 9702 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:21:04.134557 9713 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> >> W0705 16:21:09.135619 9701 monitor.cpp:186] Failed to > collect > > > > > resource > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of framework > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
