Task_Tracker_242 is not running now. I restarted the slave to recover the node. But as i saw from the work dir of Task_Tracker_242, the directory is made successfully. But there was no log in the working dir. The executor failed to start at the beginning.
Guodong On Wed, Jul 10, 2013 at 1:51 PM, Vinod Kone <vinodk...@gmail.com> wrote: > when the executor is unhealthy, the slave sends a kill command to the > executor. only when the executor terminates, does the slave transitions all > its tasks to LOST and inform the master and hence the scheduler. from your > observation, sounds like the kill signal was either being trapped by the > executor or was not being properly sent to the executor. is the > task/executor Task_Tracker_242 still running on the slave? you can check > this with 'ps'. what does the executor logs say. did it receive any signal? > > > On Tue, Jul 9, 2013 at 10:37 PM, 王国栋 <wangg...@gmail.com> wrote: > > > No log about killing Task_Tracker_242. But I can see Task_Tracker_242 is > in > > STAGING from master/slave web UI. And it is stuck. The hadoop framework > > consider that Task_Tracker_242 is launched, though it is not running. > > > > I think if slave deems Task_Tracker_242 unhealthy, it should report to > > master that this task is Lost. But I am not sure why it can not report > > this. I am trying to go through the code. > > > > Guodong > > > > > > On Wed, Jul 10, 2013 at 2:51 AM, Vinod Kone <vinodk...@gmail.com> wrote: > > > > > Hey Guodong, > > > > > > So, looks like Task_Tracker_242 did not register with the slave within > 1 > > > minute and the slave decided to kill it because it was deemed > unhealthy. > > At > > > this point the executor should've received a kill signal from the > slave. > > Do > > > you see anything of that sort in the slave or executor logs? > > > > > > > > > On Mon, Jul 8, 2013 at 11:30 PM, 王国栋 <wangg...@gmail.com> wrote: > > > > > > > Hi vinod. > > > > > > > > I am using the code from the trunk. I think the latest commit is at > Jul > > > > 1st. I will grep some master log in another mail. > > > > > > > > The Task "Task_Tracker_242" is stuck in STAGING. I think > > > "Task_Tracker_224" > > > > and "Task_Tracker_230" exit sucessfully. But it is strange that there > > > are a > > > > lot of "Fail to collect resource..." warnings. > > > > > > > > I0709 00:46:11.288698 11002 slave.cpp:739] Got assigned task > > > > Task_Tracker_242 for framework 201307040929-252063498-5050-27411-0000 > > > > I0709 00:46:11.289136 11002 slave.cpp:837] Launching task > > > Task_Tracker_242 > > > > for framework 201307040929-252063498-5050-27411-0000 > > > > I0709 00:46:11.291296 11002 paths.hpp:303] Created executor directory > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > cutor_Task_Tracker_242/runs/5c47ad99-1c78-43c8-9f27-9509f1d39c3d' > > > > I0709 00:46:11.291647 11002 slave.cpp:948] Queuing task > > > 'Task_Tracker_242' > > > > for executor executor_Task_Tracker_242 of framework > > > > '201307040929-252063498-5050-27411-0000 > > > > I0709 00:46:11.292162 11002 slave.cpp:511] Successfully attached file > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > cutor_Task_Tracker_242/runs/5c47ad99-1c78-43c8-9f27-9509f1d39c3d' > > > > W0709 00:46:12.197242 10992 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:16.100548 10994 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:17.197463 11001 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:21.101570 11002 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:22.198303 11005 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:26.102522 11002 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:27.199403 10998 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:31.103610 10998 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:32.200248 11001 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:36.104547 11004 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:37.201236 10991 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:41.105523 10997 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:42.202250 10991 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > I0709 00:46:45.283098 11002 slave.cpp:2511] Current usage 57.43%. Max > > > > allowed age: 2.279812884766227days > > > > W0709 00:46:46.106760 10994 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:47.203474 10993 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:51.107544 11006 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:52.204280 10997 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:56.108530 10995 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:46:57.205417 10997 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:47:01.109284 10997 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:47:02.206368 11002 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > I0709 00:47:05.288517 11002 slave.cpp:2463] Terminating executor > > > > executor_Task_Tracker_238 of framework > > > > 201307040929-252063498-5050-27411-0000 because it did not register > > within > > > > 1mins > > > > W0709 00:47:06.110532 11005 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:47:07.207320 10997 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > W0709 00:47:11.111778 10996 monitor.cpp:186] Failed to collect > resource > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > I0709 00:47:11.292485 10991 slave.cpp:2463] Terminating executor > > > > executor_Task_Tracker_242 of framework > > > > 201307040929-252063498-5050-27411-0000 because it did not register > > within > > > > 1mins > > > > > > > > > > > > Guodong > > > > > > > > > > > > On Tue, Jul 9, 2013 at 2:21 PM, Vinod Kone <vinodk...@gmail.com> > > wrote: > > > > > > > > > hey guodong, which of these task(s) is stuck in STAGING? also, the > > > > > corresponding master's logs would also be helpful here. also which > > > > version > > > > > of mesos are you running? > > > > > > > > > > > > > > > On Mon, Jul 8, 2013 at 11:02 PM, 王国栋 <wangg...@gmail.com> wrote: > > > > > > > > > > > It is very interesting that there are these logs. > > > > > > > > > > > > I0709 00:33:43.833853 11002 slave.cpp:996] Asked to kill task > > > > > > Task_Tracker_224 of framework > > 201307040929-252063498-5050-27411-0000 > > > > > > I0709 00:33:43.835552 11006 slave.cpp:996] Asked to kill task > > > > > > Task_Tracker_230 of framework > > 201307040929-252063498-5050-27411-0000 > > > > > > I0709 00:33:43.972771 10994 slave.cpp:1692] Handling status > update > > > > > > TASK_FINISHED (UUID: 372081cc-edf2-4183-a461-9345ab6d279c) for > task > > > > > > Task_Tracker_230 of framework > > 201307040929-252063498-5050-27411-0000 > > > > > > from executor(1)@10.47.6.21:27786 > > > > > > I0709 00:33:43.973132 10994 status_update_manager.cpp:290] > Received > > > > > status > > > > > > update TASK_FINISHED (UUID: 372081cc-edf2-4183-a461-9345ab6d279c) > > for > > > > > task > > > > > > Task_Tracker_230 of framework 201307040929-252063498-5 > > > > > > 050-27411-0000 with checkpoint=false > > > > > > I0709 00:33:43.973192 10994 status_update_manager.cpp:336] > > Forwarding > > > > > > status update TASK_FINISHED (UUID: > > > > 372081cc-edf2-4183-a461-9345ab6d279c) > > > > > > for task Task_Tracker_230 of framework 201307040929-252063498 > > > > > > -5050-27411-0000 to master@10.47.6.15:5050 > > > > > > I0709 00:33:43.973846 11005 slave.cpp:1809] Sending > acknowledgement > > > for > > > > > > status update TASK_FINISHED (UUID: > > > > 372081cc-edf2-4183-a461-9345ab6d279c) > > > > > > for task Task_Tracker_230 of framework 201307040929-2520634 > > > > > > 98-5050-27411-0000 to executor(1)@10.47.6.21:27786 > > > > > > I0709 00:33:43.974591 11000 status_update_manager.cpp:360] > Received > > > > > status > > > > > > update acknowledgement 372081cc-edf2-4183-a461-9345ab6d279c for > > task > > > > > > Task_Tracker_230 of framework 201307040929-252063498-5050-27 > > > > > > 411-0000 > > > > > > I0709 00:33:43.974652 11000 status_update_manager.cpp:481] > Cleaning > > > up > > > > > > status update stream for task Task_Tracker_230 of framework > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > I0709 00:33:44.090603 11003 slave.cpp:1692] Handling status > update > > > > > > TASK_FINISHED (UUID: 61d5775a-2375-412a-a5a4-80ab55163d88) for > task > > > > > > Task_Tracker_224 of framework > > 201307040929-252063498-5050-27411-0000 > > > > > > from executor(1)@10.47.6.21:2310 > > > > > > I0709 00:33:44.090860 11003 status_update_manager.cpp:290] > Received > > > > > status > > > > > > update TASK_FINISHED (UUID: 61d5775a-2375-412a-a5a4-80ab55163d88) > > for > > > > > task > > > > > > Task_Tracker_224 of framework 201307040929-252063498-5 > > > > > > 050-27411-0000 with checkpoint=false > > > > > > I0709 00:33:44.090973 11003 status_update_manager.cpp:336] > > Forwarding > > > > > > status update TASK_FINISHED (UUID: > > > > 61d5775a-2375-412a-a5a4-80ab55163d88) > > > > > > for task Task_Tracker_224 of framework 201307040929-252063498 > > > > > > -5050-27411-0000 to master@10.47.6.15:5050 > > > > > > I0709 00:33:44.091279 11003 slave.cpp:1809] Sending > acknowledgement > > > for > > > > > > status update TASK_FINISHED (UUID: > > > > 61d5775a-2375-412a-a5a4-80ab55163d88) > > > > > > for task Task_Tracker_224 of framework 201307040929-2520634 > > > > > > 98-5050-27411-0000 to executor(1)@10.47.6.21:2310 > > > > > > I0709 00:33:44.093286 11003 status_update_manager.cpp:360] > Received > > > > > status > > > > > > update acknowledgement 61d5775a-2375-412a-a5a4-80ab55163d88 for > > task > > > > > > Task_Tracker_224 of framework 201307040929-252063498-5050-27 > > > > > > 411-0000 > > > > > > I0709 00:33:44.093359 11003 status_update_manager.cpp:481] > Cleaning > > > up > > > > > > status update stream for task Task_Tracker_224 of framework > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > I0709 00:33:45.259831 10997 slave.cpp:2511] Current usage 57.44%. > > Max > > > > > > allowed age: 2.279168852469954days > > > > > > W0709 00:33:45.949470 10996 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:33:47.063181 11005 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:33:50.950412 11000 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:33:52.063576 10993 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:33:55.951427 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:33:57.064575 10998 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:34:00.952390 11003 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:34:02.065218 10998 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:34:05.953456 10995 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:34:07.066515 10995 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:34:10.954479 10998 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:34:12.067471 11005 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:34:15.955461 10996 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > W0709 00:34:17.068209 10996 monitor.cpp:186] Failed to collect > > > resource > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > > > > > > > > > > > > > > > > > > Guodong > > > > > > > > > > > > > > > > > > On Tue, Jul 9, 2013 at 1:59 PM, 王国栋 <wangg...@gmail.com> wrote: > > > > > > > > > > > > > Hi ben, > > > > > > > > > > > > > > I ran into the same issue here. > > > > > > > > > > > > > > This also happens in our hadoop framework. The slave log is > like > > > > these. > > > > > > At > > > > > > > that time, I think the work load of the node is very high. > > > > > > > > > > > > > > I0708 23:36:44.253880 11005 slave.cpp:739] Got assigned task > > > > > > > Task_Tracker_224 for framework > > > 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:44.255221 10999 gc.cpp:84] Unscheduling > > > > > > > > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000' > > > > > > > for removal > > > > > > > I0708 23:36:44.256206 11001 slave.cpp:837] Launching task > > > > > > Task_Tracker_224 > > > > > > > for framework 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:44.258117 11001 paths.hpp:303] Created executor > > > directory > > > > > > > > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > > > > > cutor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1' > > > > > > > I0708 23:36:44.258467 10991 process_isolator.cpp:99] Launching > > > > > > > executor_Task_Tracker_224 (cd hadoop && ./bin/mesos-executor) > in > > > > > > > > > > > > > > > > > > > > > > > > > > > > /data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/framew > > > > > > > > > > > > > > > > > > > > > > > > > > > > orks/201307040929-252063498-5050-27411-0000/executors/executor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1 > > > > > > > with resources cpus=1; mem=1280' for framework > > > > > > > 201307040929-252063498-5050-27411-0 > > > > > > > 000 > > > > > > > I0708 23:36:44.258496 11001 slave.cpp:948] Queuing task > > > > > > 'Task_Tracker_224' > > > > > > > for executor executor_Task_Tracker_224 of framework > > > > > > > '201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:44.261446 10991 process_isolator.cpp:161] Forked > > > executor > > > > > at > > > > > > > 2220 > > > > > > > I0708 23:36:44.261787 10996 slave.cpp:511] Successfully > attached > > > file > > > > > > > > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > > > > > cutor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1' > > > > > > > I0708 23:36:44.580497 10996 slave.cpp:2511] Current usage > 57.21%. > > > Max > > > > > > > allowed age: 2.295155852123924days > > > > > > > I0708 23:36:44.750393 11002 slave.cpp:1395] Got registration > for > > > > > executor > > > > > > > 'executor_Task_Tracker_224' of framework > > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:44.751095 11002 slave.cpp:1510] Flushing queued > task > > > > > > > Task_Tracker_224 for executor 'executor_Task_Tracker_224' of > > > > framework > > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:46.144317 11006 slave.cpp:1692] Handling status > > update > > > > > > > TASK_RUNNING (UUID: 364ee347-f6a2-4c7b-8702-460aa0ece579) for > > task > > > > > > > Task_Tracker_224 of framework > > > 201307040929-252063498-5050-27411-0000 > > > > f > > > > > > > rom executor(1)@10.47.6.21:2310 > > > > > > > I0708 23:36:46.144745 11006 status_update_manager.cpp:290] > > Received > > > > > > status > > > > > > > update TASK_RUNNING (UUID: > 364ee347-f6a2-4c7b-8702-460aa0ece579) > > > for > > > > > task > > > > > > > Task_Tracker_224 of framework 201307040929-252063498-50 > > > > > > > 50-27411-0000 with checkpoint=false > > > > > > > I0708 23:36:46.144821 11006 status_update_manager.cpp:450] > > Creating > > > > > > > StatusUpdate stream for task Task_Tracker_224 of framework > > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:46.145076 11006 status_update_manager.cpp:336] > > > Forwarding > > > > > > > status update TASK_RUNNING (UUID: > > > > 364ee347-f6a2-4c7b-8702-460aa0ece579) > > > > > > for > > > > > > > task Task_Tracker_224 of framework 201307040929-252063498- > > > > > > > 5050-27411-0000 to master@10.47.6.15:5050 > > > > > > > I0708 23:36:46.145882 10997 slave.cpp:1809] Sending > > acknowledgement > > > > for > > > > > > > status update TASK_RUNNING (UUID: > > > > 364ee347-f6a2-4c7b-8702-460aa0ece579) > > > > > > for > > > > > > > task Task_Tracker_224 of framework 201307040929-25206349 > > > > > > > 8-5050-27411-0000 to executor(1)@10.47.6.21:2310 > > > > > > > I0708 23:36:46.146870 10993 status_update_manager.cpp:360] > > Received > > > > > > status > > > > > > > update acknowledgement 364ee347-f6a2-4c7b-8702-460aa0ece579 for > > > task > > > > > > > Task_Tracker_224 of framework 201307040929-252063498-5050-27 > > > > > > > 411-0000 > > > > > > > I0708 23:36:50.258347 11005 slave.cpp:739] Got assigned task > > > > > > > Task_Tracker_230 for framework > > > 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:50.259472 11005 slave.cpp:837] Launching task > > > > > > Task_Tracker_230 > > > > > > > for framework 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:50.261641 11005 paths.hpp:303] Created executor > > > directory > > > > > > > > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > > > > > cutor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd' > > > > > > > I0708 23:36:50.262265 11005 slave.cpp:948] Queuing task > > > > > > 'Task_Tracker_230' > > > > > > > for executor executor_Task_Tracker_230 of framework > > > > > > > '201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:50.262557 11005 process_isolator.cpp:99] Launching > > > > > > > executor_Task_Tracker_230 (cd hadoop && ./bin/mesos-executor) > in > > > > > > > > > > > > > > > > > > > > > > > > > > > > /data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/framew > > > > > > > > > > > > > > > > > > > > > > > > > > > > orks/201307040929-252063498-5050-27411-0000/executors/executor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd > > > > > > > with resources cpus=1; mem=1280' for framework > > > > > > > 201307040929-252063498-5050-27411-0 > > > > > > > 000 > > > > > > > I0708 23:36:50.265396 10999 slave.cpp:511] Successfully > attached > > > file > > > > > > > > > > > > > > > > > > > > > > > > > > > > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > > > > > > > > cutor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd' > > > > > > > I0708 23:36:50.265419 11005 process_isolator.cpp:161] Forked > > > executor > > > > > at > > > > > > > 2851 > > > > > > > I0708 23:36:50.835607 10995 slave.cpp:1395] Got registration > for > > > > > executor > > > > > > > 'executor_Task_Tracker_230' of framework > > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:50.836174 10995 slave.cpp:1510] Flushing queued > task > > > > > > > Task_Tracker_230 for executor 'executor_Task_Tracker_230' of > > > > framework > > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:54.617856 10994 slave.cpp:1692] Handling status > > update > > > > > > > TASK_RUNNING (UUID: 7753252d-c90b-4b0d-adca-7c97f38f692e) for > > task > > > > > > > Task_Tracker_230 of framework > > > 201307040929-252063498-5050-27411-0000 > > > > f > > > > > > > rom executor(1)@10.47.6.21:27786 > > > > > > > I0708 23:36:54.618275 10994 status_update_manager.cpp:290] > > Received > > > > > > status > > > > > > > update TASK_RUNNING (UUID: > 7753252d-c90b-4b0d-adca-7c97f38f692e) > > > for > > > > > task > > > > > > > Task_Tracker_230 of framework 201307040929-252063498-50 > > > > > > > 50-27411-0000 with checkpoint=false > > > > > > > I0708 23:36:54.618326 10994 status_update_manager.cpp:450] > > Creating > > > > > > > StatusUpdate stream for task Task_Tracker_230 of framework > > > > > > > 201307040929-252063498-5050-27411-0000 > > > > > > > I0708 23:36:54.618443 10994 status_update_manager.cpp:336] > > > Forwarding > > > > > > > status update TASK_RUNNING (UUID: > > > > 7753252d-c90b-4b0d-adca-7c97f38f692e) > > > > > > for > > > > > > > task Task_Tracker_230 of framework 201307040929-252063498- > > > > > > > 5050-27411-0000 to master@10.47.6.15:5050 > > > > > > > I0708 23:36:54.619137 10994 slave.cpp:1809] Sending > > acknowledgement > > > > for > > > > > > > status update TASK_RUNNING (UUID: > > > > 7753252d-c90b-4b0d-adca-7c97f38f692e) > > > > > > for > > > > > > > task Task_Tracker_230 of framework 201307040929-25206349 > > > > > > > 8-5050-27411-0000 to executor(1)@10.47.6.21:27786 > > > > > > > I0708 23:36:54.637682 10994 status_update_manager.cpp:360] > > Received > > > > > > status > > > > > > > update acknowledgement 7753252d-c90b-4b0d-adca-7c97f38f692e for > > > task > > > > > > > Task_Tracker_230 of framework 201307040929-252063498-5050-27 > > > > > > > 411-0000 > > > > > > > I0708 23:37:44.583014 11002 slave.cpp:2511] Current usage > 57.23%. > > > Max > > > > > > > allowed age: 2.293704423241597days > > > > > > > I0708 23:38:44.585233 11003 slave.cpp:2511] Current usage > 57.23%. > > > Max > > > > > > > allowed age: 2.293703916528542days > > > > > > > I0708 23:39:44.599442 11006 slave.cpp:2511] Current usage > 57.23%. > > > Max > > > > > > > allowed age: 2.293639867998055days > > > > > > > I0708 23:40:44.603996 10997 slave.cpp:2511] Current usage > 57.24%. > > > Max > > > > > > > allowed age: 2.292921551567535days > > > > > > > I0708 23:41:44.608608 11006 slave.cpp:2511] Current usage > 57.26%. > > > Max > > > > > > > allowed age: 2.291521098018820days > > > > > > > I0708 23:42:44.609956 10992 slave.cpp:2511] Current usage > 57.23%. > > > Max > > > > > > > allowed age: 2.293668041244063days > > > > > > > I0708 23:43:44.682621 11000 slave.cpp:2511] Current usage > 57.24%. > > > Max > > > > > > > allowed age: 2.292935638190544days > > > > > > > I0708 23:44:44.684306 10993 slave.cpp:2511] Current usage > 57.24%. > > > Max > > > > > > > allowed age: 2.292916079066516days > > > > > > > I0708 23:45:44.686172 11001 slave.cpp:2511] Current usage > 57.26%. > > > Max > > > > > > > allowed age: 2.291485324076945days > > > > > > > I0708 23:46:44.699095 10995 slave.cpp:2511] Current usage > 57.23%. > > > Max > > > > > > > allowed age: 2.293641894850289days > > > > > > > I0708 23:47:44.721156 10998 slave.cpp:2511] Current usage > 57.23%. > > > Max > > > > > > > allowed age: 2.293629429709074days > > > > > > > I0708 23:48:44.779767 10992 slave.cpp:2511] Current usage > 57.24%. > > > Max > > > > > > > allowed age: 2.293525350847025days > > > > > > > I0708 23:49:44.812389 11004 slave.cpp:2511] Current usage > 57.24%. > > > Max > > > > > > > allowed age: 2.292909289111539days > > > > > > > I0708 23:50:44.814146 10999 slave.cpp:2511] Current usage > 57.27%. > > > Max > > > > > > > allowed age: 2.291438098419977days > > > > > > > I0708 23:51:44.814877 11005 slave.cpp:2511] Current usage > 57.23%. > > > Max > > > > > > > allowed age: 2.293635104895313days > > > > > > > I0708 23:52:44.818620 10998 slave.cpp:2511] Current usage > 57.24%. > > > Max > > > > > > > allowed age: 2.292983775931019days > > > > > > > I0708 23:53:44.829911 10997 slave.cpp:2511] Current usage > 57.33%. > > > Max > > > > > > > allowed age: 2.286910009194236days > > > > > > > I0708 23:54:44.831307 10999 slave.cpp:2511] Current usage > 57.33%. > > > Max > > > > > > > allowed age: 2.286909502481169days > > > > > > > I0708 23:55:44.902858 10994 slave.cpp:2511] Current usage > 57.37%. > > > Max > > > > > > > allowed age: 2.284414244700093days > > > > > > > I0708 23:56:44.905398 11002 slave.cpp:2511] Current usage > 57.42%. > > > Max > > > > > > > allowed age: 2.280636901540567days > > > > > > > I0708 23:57:44.933673 10991 slave.cpp:2511] Current usage > 57.44%. > > > Max > > > > > > > allowed age: 2.279481899796968days > > > > > > > I0708 23:58:44.934840 11004 slave.cpp:2511] Current usage > 57.48%. > > > Max > > > > > > > allowed age: 2.276566475548496days > > > > > > > I0708 23:59:44.936063 11001 slave.cpp:2511] Current usage > 57.49%. > > > Max > > > > > > > allowed age: 2.275690368671817days > > > > > > > I0709 00:00:44.937433 11004 slave.cpp:2511] Current usage > 57.50%. > > > Max > > > > > > > allowed age: 2.275057180034989days > > > > > > > I0709 00:01:44.938940 11001 slave.cpp:2511] Current usage > 57.51%. > > > Max > > > > > > > allowed age: 2.273999467198449days > > > > > > > I0709 00:02:44.955103 10996 slave.cpp:2511] Current usage > 57.52%. > > > Max > > > > > > > allowed age: 2.273472384275891days > > > > > > > I0709 00:03:44.956354 10993 slave.cpp:2511] Current usage > 57.39%. > > > Max > > > > > > > allowed age: 2.282894612240220days > > > > > > > I0709 00:04:44.957926 10997 slave.cpp:2511] Current usage > 57.40%. > > > Max > > > > > > > allowed age: 2.281966516603831days > > > > > > > I0709 00:05:44.969205 10996 slave.cpp:2511] Current usage > 57.40%. > > > Max > > > > > > > allowed age: 2.281962260214144days > > > > > > > I0709 00:06:44.969987 10992 slave.cpp:2511] Current usage > 57.40%. > > > Max > > > > > > > allowed age: 2.281791801941551days > > > > > > > I0709 00:07:44.977504 11004 slave.cpp:2511] Current usage > 57.40%. > > > Max > > > > > > > allowed age: 2.281715288269849days > > > > > > > I0709 00:08:44.982868 10998 slave.cpp:2511] Current usage > 57.40%. > > > Max > > > > > > > allowed age: 2.281699782850289days > > > > > > > I0709 00:09:44.997082 11000 slave.cpp:2511] Current usage > 57.42%. > > > Max > > > > > > > allowed age: 2.280776044946192days > > > > > > > I0709 00:10:44.998754 10994 slave.cpp:2511] Current usage > 57.42%. > > > Max > > > > > > > allowed age: 2.280772193926956days > > > > > > > I0709 00:11:44.999949 11002 slave.cpp:2511] Current usage > 57.44%. > > > Max > > > > > > > allowed age: 2.279204525069213days > > > > > > > I0709 00:12:45.001539 10995 slave.cpp:2511] Current usage > 57.47%. > > > Max > > > > > > > allowed age: 2.277132676719109days > > > > > > > I0709 00:13:45.002728 10992 slave.cpp:2511] Current usage > 57.43%. > > > Max > > > > > > > allowed age: 2.280012428368322days > > > > > > > I0709 00:14:45.009699 10998 slave.cpp:2511] Current usage > 57.48%. > > > Max > > > > > > > allowed age: 2.276733690857512days > > > > > > > I0709 00:15:45.013483 10996 slave.cpp:2511] Current usage > 57.53%. > > > Max > > > > > > > allowed age: 2.272715152282546days > > > > > > > I0709 00:16:45.015496 10998 slave.cpp:2511] Current usage > 57.57%. > > > Max > > > > > > > allowed age: 2.270354274804352days > > > > > > > I0709 00:17:45.016628 11000 slave.cpp:2511] Current usage > 57.62%. > > > Max > > > > > > > allowed age: 2.266927678423322days > > > > > > > I0709 00:18:45.032670 11002 slave.cpp:2511] Current usage > 57.65%. > > > Max > > > > > > > allowed age: 2.264218182361482days > > > > > > > I0709 00:19:45.043442 10998 slave.cpp:2511] Current usage > 57.69%. > > > Max > > > > > > > allowed age: 2.261509598383137days > > > > > > > I0709 00:20:45.080648 10992 slave.cpp:2511] Current usage > 57.72%. > > > Max > > > > > > > allowed age: 2.259379478031400days > > > > > > > I0709 00:21:45.081632 10995 slave.cpp:2511] Current usage > 57.77%. > > > Max > > > > > > > allowed age: 2.255819920144039days > > > > > > > I0709 00:22:45.082593 11005 slave.cpp:2511] Current usage > 57.81%. > > > Max > > > > > > > allowed age: 2.253314528101817days > > > > > > > I0709 00:23:45.193588 10997 slave.cpp:2511] Current usage > 57.85%. > > > Max > > > > > > > allowed age: 2.250524870034248days > > > > > > > I0709 00:24:45.220617 10994 slave.cpp:2511] Current usage > 57.90%. > > > Max > > > > > > > allowed age: 2.246784618270532days > > > > > > > I0709 00:25:45.241602 10992 slave.cpp:2511] Current usage > 57.97%. > > > Max > > > > > > > allowed age: 2.242399422127049days > > > > > > > I0709 00:26:45.248977 11000 slave.cpp:2511] Current usage > 58.00%. > > > Max > > > > > > > allowed age: 2.240250654734792days > > > > > > > I0709 00:27:45.250953 10993 slave.cpp:2511] Current usage > 57.99%. > > > Max > > > > > > > allowed age: 2.240516983117894days > > > > > > > I0709 00:28:45.252694 10996 slave.cpp:2511] Current usage > 58.06%. > > > Max > > > > > > > allowed age: 2.235834143724352days > > > > > > > I0709 00:29:45.254992 11003 slave.cpp:2511] Current usage > 58.10%. > > > Max > > > > > > > allowed age: 2.233297436815162days > > > > > > > W0709 00:30:06.753098 10999 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:10.715373 10996 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:11.754446 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:15.719880 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:16.755473 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:20.720330 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:21.766019 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:25.721364 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:26.768874 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:30.722605 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:31.770354 11003 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:35.724455 10992 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:36.788751 10992 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:40.745380 10992 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_224' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > W0709 00:30:41.789358 10992 monitor.cpp:186] Failed to collect > > > > resource > > > > > > > usage for executor 'executor_Task_Tracker_230' of framework > > > > > > > '201307040929-252063498-5050-27411-0000': Future discarded > > > > > > > I0709 00:30:45.256590 11004 slave.cpp:2511] Current usage > 58.11%. > > > Max > > > > > > > allowed age: 2.232469873049410days > > > > > > > > > > > > > > > > > > > > > Guodong > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 9, 2013 at 4:55 AM, Benjamin Mahler < > > > > > > benjamin.mah...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > >> Are these the un-edited logs? I'm expecting to see some logs > > from > > > > the > > > > > > >> process_isolator or cgroups_isolator in there. > > > > > > >> > > > > > > >> > > > > > > >> On Fri, Jul 5, 2013 at 2:38 PM, Brenden Matthews < > > > > > > >> brenden.matth...@airbedandbreakfast.com> wrote: > > > > > > >> > > > > > > >> > Hey guys, > > > > > > >> > > > > > > > >> > I'm currently having a problem where tasks will get stuck in > > the > > > > > > staging > > > > > > >> > state, though according to the logs they should have been > > > > > terminated. > > > > > > >> They > > > > > > >> > hang indefinitely, or until I restart the slave. Below is a > > > > > > screenshot > > > > > > >> + > > > > > > >> > logs. Also interesting is the 'Failed to collect resource > > usage > > > > > ...' > > > > > > >> > messages. > > > > > > >> > > > > > > > >> > [image: Inline image 2] > > > > > > >> > > > > > > > >> > I0705 16:19:51.551512 9706 slave.cpp:739] Got assigned task > > > > > > >> >> > > > ct:1373041190990:0:add_latest_reservation_survey_events_partitio > > > > > > >> >> n for framework chronos > > > > > > >> >> I0705 16:19:51.552150 9706 slave.cpp:837] Launching task > > > > > > >> >> > > > > ct:1373041190990:0:add_latest_reservation_survey_events_partition f > > > > > > >> >> or framework chronos > > > > > > >> >> I0705 16:19:51.553956 9706 paths.hpp:303] Created executor > > > > > directory > > > > > > >> >> '/tmp/mesos/slaves/201307030043-2037266954-5050-15277-1 > > > > > > >> >> > > > > > > >> >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > 517/frameworks/chronos/executors/ct:1373041190990:0:add_latest_reservation_survey_events_partition/runs/611ba128-557f-4b5e-8c > > > > > > >> >> f2-4d1ce60d618f' > > > > > > >> >> I0705 16:19:51.554576 9706 slave.cpp:948] Queuing task > > > > > > >> >> > > > > > > 'ct:1373041190990:0:add_latest_reservation_survey_events_partition' f > > > > > > >> >> or executor > > > > > > >> >> > > > ct:1373041190990:0:add_latest_reservation_survey_events_partition > > > > > of > > > > > > >> >> framework 'c > > > > > > >> >> hronos > > > > > > >> >> I0705 16:19:51.555027 9706 slave.cpp:511] Successfully > > > attached > > > > > file > > > > > > >> >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > '/tmp/mesos/slaves/201307030043-2037266954-5050-15277-1517/frameworks/chronos/executors/ct:1373041190990:0:add_latest_reservation_survey_events_partition/runs/611ba128-557f-4b5e-8cf2-4d1ce60d618f' > > > > > > >> >> I0705 16:19:54.048754 9724 slave.cpp:2530] Current usage > > > 42.18%. > > > > > Max > > > > > > >> >> allowed age: 22.955009563956388hrs > > > > > > >> >> W0705 16:19:54.108963 9724 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:19:59.110787 9729 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:04.112406 9704 monitor.cpp:186] Failed to > > collect > > > > > > >> resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:09.114367 9705 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:14.116312 9706 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:19.118370 9699 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:24.120311 9701 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:29.122355 9700 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:34.123443 9722 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:39.125660 9718 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:44.127464 9724 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:49.129385 9725 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> I0705 16:20:51.555174 9703 slave.cpp:2482] Terminating > > > executor > > > > > > >> >> > > > ct:1373041190990:0:add_latest_reservation_survey_events_partition > > > > > of > > > > > > >> >> framework chronos because it did not register within 1mins > > > > > > >> >> I0705 16:20:54.050434 9717 slave.cpp:2530] Current usage > > > 42.18%. > > > > > Max > > > > > > >> >> allowed age: 22.955009342481944hrs > > > > > > >> >> W0705 16:20:54.130730 9699 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:20:59.132472 9702 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:21:04.134557 9713 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> >> W0705 16:21:09.135619 9701 monitor.cpp:186] Failed to > > collect > > > > > > resource > > > > > > >> >> usage for executor 'executor_Task_Tracker_8023' of > framework > > > > > > >> >> '201307030043-2037266954-5050-15277-0006': Future discarded > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
