It is very interesting that there are these logs. I0709 00:33:43.833853 11002 slave.cpp:996] Asked to kill task Task_Tracker_224 of framework 201307040929-252063498-5050-27411-0000 I0709 00:33:43.835552 11006 slave.cpp:996] Asked to kill task Task_Tracker_230 of framework 201307040929-252063498-5050-27411-0000 I0709 00:33:43.972771 10994 slave.cpp:1692] Handling status update TASK_FINISHED (UUID: 372081cc-edf2-4183-a461-9345ab6d279c) for task Task_Tracker_230 of framework 201307040929-252063498-5050-27411-0000 from executor(1)@10.47.6.21:27786 I0709 00:33:43.973132 10994 status_update_manager.cpp:290] Received status update TASK_FINISHED (UUID: 372081cc-edf2-4183-a461-9345ab6d279c) for task Task_Tracker_230 of framework 201307040929-252063498-5 050-27411-0000 with checkpoint=false I0709 00:33:43.973192 10994 status_update_manager.cpp:336] Forwarding status update TASK_FINISHED (UUID: 372081cc-edf2-4183-a461-9345ab6d279c) for task Task_Tracker_230 of framework 201307040929-252063498 -5050-27411-0000 to master@10.47.6.15:5050 I0709 00:33:43.973846 11005 slave.cpp:1809] Sending acknowledgement for status update TASK_FINISHED (UUID: 372081cc-edf2-4183-a461-9345ab6d279c) for task Task_Tracker_230 of framework 201307040929-2520634 98-5050-27411-0000 to executor(1)@10.47.6.21:27786 I0709 00:33:43.974591 11000 status_update_manager.cpp:360] Received status update acknowledgement 372081cc-edf2-4183-a461-9345ab6d279c for task Task_Tracker_230 of framework 201307040929-252063498-5050-27 411-0000 I0709 00:33:43.974652 11000 status_update_manager.cpp:481] Cleaning up status update stream for task Task_Tracker_230 of framework 201307040929-252063498-5050-27411-0000 I0709 00:33:44.090603 11003 slave.cpp:1692] Handling status update TASK_FINISHED (UUID: 61d5775a-2375-412a-a5a4-80ab55163d88) for task Task_Tracker_224 of framework 201307040929-252063498-5050-27411-0000 from executor(1)@10.47.6.21:2310 I0709 00:33:44.090860 11003 status_update_manager.cpp:290] Received status update TASK_FINISHED (UUID: 61d5775a-2375-412a-a5a4-80ab55163d88) for task Task_Tracker_224 of framework 201307040929-252063498-5 050-27411-0000 with checkpoint=false I0709 00:33:44.090973 11003 status_update_manager.cpp:336] Forwarding status update TASK_FINISHED (UUID: 61d5775a-2375-412a-a5a4-80ab55163d88) for task Task_Tracker_224 of framework 201307040929-252063498 -5050-27411-0000 to master@10.47.6.15:5050 I0709 00:33:44.091279 11003 slave.cpp:1809] Sending acknowledgement for status update TASK_FINISHED (UUID: 61d5775a-2375-412a-a5a4-80ab55163d88) for task Task_Tracker_224 of framework 201307040929-2520634 98-5050-27411-0000 to executor(1)@10.47.6.21:2310 I0709 00:33:44.093286 11003 status_update_manager.cpp:360] Received status update acknowledgement 61d5775a-2375-412a-a5a4-80ab55163d88 for task Task_Tracker_224 of framework 201307040929-252063498-5050-27 411-0000 I0709 00:33:44.093359 11003 status_update_manager.cpp:481] Cleaning up status update stream for task Task_Tracker_224 of framework 201307040929-252063498-5050-27411-0000 I0709 00:33:45.259831 10997 slave.cpp:2511] Current usage 57.44%. Max allowed age: 2.279168852469954days W0709 00:33:45.949470 10996 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_224' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:33:47.063181 11005 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_230' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:33:50.950412 11000 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_224' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:33:52.063576 10993 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_230' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:33:55.951427 11003 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_224' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:33:57.064575 10998 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_230' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:34:00.952390 11003 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_224' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:34:02.065218 10998 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_230' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:34:05.953456 10995 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_224' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:34:07.066515 10995 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_230' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:34:10.954479 10998 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_224' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:34:12.067471 11005 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_230' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:34:15.955461 10996 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_224' of framework '201307040929-252063498-5050-27411-0000': Future discarded W0709 00:34:17.068209 10996 monitor.cpp:186] Failed to collect resource usage for executor 'executor_Task_Tracker_230' of framework '201307040929-252063498-5050-27411-0000': Future discarded
Guodong On Tue, Jul 9, 2013 at 1:59 PM, 王国栋 <wangg...@gmail.com> wrote: > Hi ben, > > I ran into the same issue here. > > This also happens in our hadoop framework. The slave log is like these. At > that time, I think the work load of the node is very high. > > I0708 23:36:44.253880 11005 slave.cpp:739] Got assigned task > Task_Tracker_224 for framework 201307040929-252063498-5050-27411-0000 > I0708 23:36:44.255221 10999 gc.cpp:84] Unscheduling > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000' > for removal > I0708 23:36:44.256206 11001 slave.cpp:837] Launching task Task_Tracker_224 > for framework 201307040929-252063498-5050-27411-0000 > I0708 23:36:44.258117 11001 paths.hpp:303] Created executor directory > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > cutor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1' > I0708 23:36:44.258467 10991 process_isolator.cpp:99] Launching > executor_Task_Tracker_224 (cd hadoop && ./bin/mesos-executor) in > /data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/framew > orks/201307040929-252063498-5050-27411-0000/executors/executor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1 > with resources cpus=1; mem=1280' for framework > 201307040929-252063498-5050-27411-0 > 000 > I0708 23:36:44.258496 11001 slave.cpp:948] Queuing task 'Task_Tracker_224' > for executor executor_Task_Tracker_224 of framework > '201307040929-252063498-5050-27411-0000 > I0708 23:36:44.261446 10991 process_isolator.cpp:161] Forked executor at > 2220 > I0708 23:36:44.261787 10996 slave.cpp:511] Successfully attached file > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > cutor_Task_Tracker_224/runs/953d3565-424c-4ab3-9926-a3fa71042bf1' > I0708 23:36:44.580497 10996 slave.cpp:2511] Current usage 57.21%. Max > allowed age: 2.295155852123924days > I0708 23:36:44.750393 11002 slave.cpp:1395] Got registration for executor > 'executor_Task_Tracker_224' of framework > 201307040929-252063498-5050-27411-0000 > I0708 23:36:44.751095 11002 slave.cpp:1510] Flushing queued task > Task_Tracker_224 for executor 'executor_Task_Tracker_224' of framework > 201307040929-252063498-5050-27411-0000 > I0708 23:36:46.144317 11006 slave.cpp:1692] Handling status update > TASK_RUNNING (UUID: 364ee347-f6a2-4c7b-8702-460aa0ece579) for task > Task_Tracker_224 of framework 201307040929-252063498-5050-27411-0000 f > rom executor(1)@10.47.6.21:2310 > I0708 23:36:46.144745 11006 status_update_manager.cpp:290] Received status > update TASK_RUNNING (UUID: 364ee347-f6a2-4c7b-8702-460aa0ece579) for task > Task_Tracker_224 of framework 201307040929-252063498-50 > 50-27411-0000 with checkpoint=false > I0708 23:36:46.144821 11006 status_update_manager.cpp:450] Creating > StatusUpdate stream for task Task_Tracker_224 of framework > 201307040929-252063498-5050-27411-0000 > I0708 23:36:46.145076 11006 status_update_manager.cpp:336] Forwarding > status update TASK_RUNNING (UUID: 364ee347-f6a2-4c7b-8702-460aa0ece579) for > task Task_Tracker_224 of framework 201307040929-252063498- > 5050-27411-0000 to master@10.47.6.15:5050 > I0708 23:36:46.145882 10997 slave.cpp:1809] Sending acknowledgement for > status update TASK_RUNNING (UUID: 364ee347-f6a2-4c7b-8702-460aa0ece579) for > task Task_Tracker_224 of framework 201307040929-25206349 > 8-5050-27411-0000 to executor(1)@10.47.6.21:2310 > I0708 23:36:46.146870 10993 status_update_manager.cpp:360] Received status > update acknowledgement 364ee347-f6a2-4c7b-8702-460aa0ece579 for task > Task_Tracker_224 of framework 201307040929-252063498-5050-27 > 411-0000 > I0708 23:36:50.258347 11005 slave.cpp:739] Got assigned task > Task_Tracker_230 for framework 201307040929-252063498-5050-27411-0000 > I0708 23:36:50.259472 11005 slave.cpp:837] Launching task Task_Tracker_230 > for framework 201307040929-252063498-5050-27411-0000 > I0708 23:36:50.261641 11005 paths.hpp:303] Created executor directory > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > cutor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd' > I0708 23:36:50.262265 11005 slave.cpp:948] Queuing task 'Task_Tracker_230' > for executor executor_Task_Tracker_230 of framework > '201307040929-252063498-5050-27411-0000 > I0708 23:36:50.262557 11005 process_isolator.cpp:99] Launching > executor_Task_Tracker_230 (cd hadoop && ./bin/mesos-executor) in > /data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/framew > orks/201307040929-252063498-5050-27411-0000/executors/executor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd > with resources cpus=1; mem=1280' for framework > 201307040929-252063498-5050-27411-0 > 000 > I0708 23:36:50.265396 10999 slave.cpp:511] Successfully attached file > '/data/mesos-slave-work-dir//slaves/201307041648-252063498-5050-8038-5/frameworks/201307040929-252063498-5050-27411-0000/executors/exe > cutor_Task_Tracker_230/runs/1dc87acc-d090-469f-ba30-0477139ee7fd' > I0708 23:36:50.265419 11005 process_isolator.cpp:161] Forked executor at > 2851 > I0708 23:36:50.835607 10995 slave.cpp:1395] Got registration for executor > 'executor_Task_Tracker_230' of framework > 201307040929-252063498-5050-27411-0000 > I0708 23:36:50.836174 10995 slave.cpp:1510] Flushing queued task > Task_Tracker_230 for executor 'executor_Task_Tracker_230' of framework > 201307040929-252063498-5050-27411-0000 > I0708 23:36:54.617856 10994 slave.cpp:1692] Handling status update > TASK_RUNNING (UUID: 7753252d-c90b-4b0d-adca-7c97f38f692e) for task > Task_Tracker_230 of framework 201307040929-252063498-5050-27411-0000 f > rom executor(1)@10.47.6.21:27786 > I0708 23:36:54.618275 10994 status_update_manager.cpp:290] Received status > update TASK_RUNNING (UUID: 7753252d-c90b-4b0d-adca-7c97f38f692e) for task > Task_Tracker_230 of framework 201307040929-252063498-50 > 50-27411-0000 with checkpoint=false > I0708 23:36:54.618326 10994 status_update_manager.cpp:450] Creating > StatusUpdate stream for task Task_Tracker_230 of framework > 201307040929-252063498-5050-27411-0000 > I0708 23:36:54.618443 10994 status_update_manager.cpp:336] Forwarding > status update TASK_RUNNING (UUID: 7753252d-c90b-4b0d-adca-7c97f38f692e) for > task Task_Tracker_230 of framework 201307040929-252063498- > 5050-27411-0000 to master@10.47.6.15:5050 > I0708 23:36:54.619137 10994 slave.cpp:1809] Sending acknowledgement for > status update TASK_RUNNING (UUID: 7753252d-c90b-4b0d-adca-7c97f38f692e) for > task Task_Tracker_230 of framework 201307040929-25206349 > 8-5050-27411-0000 to executor(1)@10.47.6.21:27786 > I0708 23:36:54.637682 10994 status_update_manager.cpp:360] Received status > update acknowledgement 7753252d-c90b-4b0d-adca-7c97f38f692e for task > Task_Tracker_230 of framework 201307040929-252063498-5050-27 > 411-0000 > I0708 23:37:44.583014 11002 slave.cpp:2511] Current usage 57.23%. Max > allowed age: 2.293704423241597days > I0708 23:38:44.585233 11003 slave.cpp:2511] Current usage 57.23%. Max > allowed age: 2.293703916528542days > I0708 23:39:44.599442 11006 slave.cpp:2511] Current usage 57.23%. Max > allowed age: 2.293639867998055days > I0708 23:40:44.603996 10997 slave.cpp:2511] Current usage 57.24%. Max > allowed age: 2.292921551567535days > I0708 23:41:44.608608 11006 slave.cpp:2511] Current usage 57.26%. Max > allowed age: 2.291521098018820days > I0708 23:42:44.609956 10992 slave.cpp:2511] Current usage 57.23%. Max > allowed age: 2.293668041244063days > I0708 23:43:44.682621 11000 slave.cpp:2511] Current usage 57.24%. Max > allowed age: 2.292935638190544days > I0708 23:44:44.684306 10993 slave.cpp:2511] Current usage 57.24%. Max > allowed age: 2.292916079066516days > I0708 23:45:44.686172 11001 slave.cpp:2511] Current usage 57.26%. Max > allowed age: 2.291485324076945days > I0708 23:46:44.699095 10995 slave.cpp:2511] Current usage 57.23%. Max > allowed age: 2.293641894850289days > I0708 23:47:44.721156 10998 slave.cpp:2511] Current usage 57.23%. Max > allowed age: 2.293629429709074days > I0708 23:48:44.779767 10992 slave.cpp:2511] Current usage 57.24%. Max > allowed age: 2.293525350847025days > I0708 23:49:44.812389 11004 slave.cpp:2511] Current usage 57.24%. Max > allowed age: 2.292909289111539days > I0708 23:50:44.814146 10999 slave.cpp:2511] Current usage 57.27%. Max > allowed age: 2.291438098419977days > I0708 23:51:44.814877 11005 slave.cpp:2511] Current usage 57.23%. Max > allowed age: 2.293635104895313days > I0708 23:52:44.818620 10998 slave.cpp:2511] Current usage 57.24%. Max > allowed age: 2.292983775931019days > I0708 23:53:44.829911 10997 slave.cpp:2511] Current usage 57.33%. Max > allowed age: 2.286910009194236days > I0708 23:54:44.831307 10999 slave.cpp:2511] Current usage 57.33%. Max > allowed age: 2.286909502481169days > I0708 23:55:44.902858 10994 slave.cpp:2511] Current usage 57.37%. Max > allowed age: 2.284414244700093days > I0708 23:56:44.905398 11002 slave.cpp:2511] Current usage 57.42%. Max > allowed age: 2.280636901540567days > I0708 23:57:44.933673 10991 slave.cpp:2511] Current usage 57.44%. Max > allowed age: 2.279481899796968days > I0708 23:58:44.934840 11004 slave.cpp:2511] Current usage 57.48%. Max > allowed age: 2.276566475548496days > I0708 23:59:44.936063 11001 slave.cpp:2511] Current usage 57.49%. Max > allowed age: 2.275690368671817days > I0709 00:00:44.937433 11004 slave.cpp:2511] Current usage 57.50%. Max > allowed age: 2.275057180034989days > I0709 00:01:44.938940 11001 slave.cpp:2511] Current usage 57.51%. Max > allowed age: 2.273999467198449days > I0709 00:02:44.955103 10996 slave.cpp:2511] Current usage 57.52%. Max > allowed age: 2.273472384275891days > I0709 00:03:44.956354 10993 slave.cpp:2511] Current usage 57.39%. Max > allowed age: 2.282894612240220days > I0709 00:04:44.957926 10997 slave.cpp:2511] Current usage 57.40%. Max > allowed age: 2.281966516603831days > I0709 00:05:44.969205 10996 slave.cpp:2511] Current usage 57.40%. Max > allowed age: 2.281962260214144days > I0709 00:06:44.969987 10992 slave.cpp:2511] Current usage 57.40%. Max > allowed age: 2.281791801941551days > I0709 00:07:44.977504 11004 slave.cpp:2511] Current usage 57.40%. Max > allowed age: 2.281715288269849days > I0709 00:08:44.982868 10998 slave.cpp:2511] Current usage 57.40%. Max > allowed age: 2.281699782850289days > I0709 00:09:44.997082 11000 slave.cpp:2511] Current usage 57.42%. Max > allowed age: 2.280776044946192days > I0709 00:10:44.998754 10994 slave.cpp:2511] Current usage 57.42%. Max > allowed age: 2.280772193926956days > I0709 00:11:44.999949 11002 slave.cpp:2511] Current usage 57.44%. Max > allowed age: 2.279204525069213days > I0709 00:12:45.001539 10995 slave.cpp:2511] Current usage 57.47%. Max > allowed age: 2.277132676719109days > I0709 00:13:45.002728 10992 slave.cpp:2511] Current usage 57.43%. Max > allowed age: 2.280012428368322days > I0709 00:14:45.009699 10998 slave.cpp:2511] Current usage 57.48%. Max > allowed age: 2.276733690857512days > I0709 00:15:45.013483 10996 slave.cpp:2511] Current usage 57.53%. Max > allowed age: 2.272715152282546days > I0709 00:16:45.015496 10998 slave.cpp:2511] Current usage 57.57%. Max > allowed age: 2.270354274804352days > I0709 00:17:45.016628 11000 slave.cpp:2511] Current usage 57.62%. Max > allowed age: 2.266927678423322days > I0709 00:18:45.032670 11002 slave.cpp:2511] Current usage 57.65%. Max > allowed age: 2.264218182361482days > I0709 00:19:45.043442 10998 slave.cpp:2511] Current usage 57.69%. Max > allowed age: 2.261509598383137days > I0709 00:20:45.080648 10992 slave.cpp:2511] Current usage 57.72%. Max > allowed age: 2.259379478031400days > I0709 00:21:45.081632 10995 slave.cpp:2511] Current usage 57.77%. Max > allowed age: 2.255819920144039days > I0709 00:22:45.082593 11005 slave.cpp:2511] Current usage 57.81%. Max > allowed age: 2.253314528101817days > I0709 00:23:45.193588 10997 slave.cpp:2511] Current usage 57.85%. Max > allowed age: 2.250524870034248days > I0709 00:24:45.220617 10994 slave.cpp:2511] Current usage 57.90%. Max > allowed age: 2.246784618270532days > I0709 00:25:45.241602 10992 slave.cpp:2511] Current usage 57.97%. Max > allowed age: 2.242399422127049days > I0709 00:26:45.248977 11000 slave.cpp:2511] Current usage 58.00%. Max > allowed age: 2.240250654734792days > I0709 00:27:45.250953 10993 slave.cpp:2511] Current usage 57.99%. Max > allowed age: 2.240516983117894days > I0709 00:28:45.252694 10996 slave.cpp:2511] Current usage 58.06%. Max > allowed age: 2.235834143724352days > I0709 00:29:45.254992 11003 slave.cpp:2511] Current usage 58.10%. Max > allowed age: 2.233297436815162days > W0709 00:30:06.753098 10999 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_230' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:10.715373 10996 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_224' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:11.754446 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_230' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:15.719880 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_224' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:16.755473 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_230' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:20.720330 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_224' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:21.766019 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_230' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:25.721364 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_224' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:26.768874 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_230' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:30.722605 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_224' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:31.770354 11003 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_230' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:35.724455 10992 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_224' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:36.788751 10992 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_230' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:40.745380 10992 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_224' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > W0709 00:30:41.789358 10992 monitor.cpp:186] Failed to collect resource > usage for executor 'executor_Task_Tracker_230' of framework > '201307040929-252063498-5050-27411-0000': Future discarded > I0709 00:30:45.256590 11004 slave.cpp:2511] Current usage 58.11%. Max > allowed age: 2.232469873049410days > > > Guodong > > > On Tue, Jul 9, 2013 at 4:55 AM, Benjamin Mahler <benjamin.mah...@gmail.com > > wrote: > >> Are these the un-edited logs? I'm expecting to see some logs from the >> process_isolator or cgroups_isolator in there. >> >> >> On Fri, Jul 5, 2013 at 2:38 PM, Brenden Matthews < >> brenden.matth...@airbedandbreakfast.com> wrote: >> >> > Hey guys, >> > >> > I'm currently having a problem where tasks will get stuck in the staging >> > state, though according to the logs they should have been terminated. >> They >> > hang indefinitely, or until I restart the slave. Below is a screenshot >> + >> > logs. Also interesting is the 'Failed to collect resource usage ...' >> > messages. >> > >> > [image: Inline image 2] >> > >> > I0705 16:19:51.551512 9706 slave.cpp:739] Got assigned task >> >> ct:1373041190990:0:add_latest_reservation_survey_events_partitio >> >> n for framework chronos >> >> I0705 16:19:51.552150 9706 slave.cpp:837] Launching task >> >> ct:1373041190990:0:add_latest_reservation_survey_events_partition f >> >> or framework chronos >> >> I0705 16:19:51.553956 9706 paths.hpp:303] Created executor directory >> >> '/tmp/mesos/slaves/201307030043-2037266954-5050-15277-1 >> >> >> >> >> 517/frameworks/chronos/executors/ct:1373041190990:0:add_latest_reservation_survey_events_partition/runs/611ba128-557f-4b5e-8c >> >> f2-4d1ce60d618f' >> >> I0705 16:19:51.554576 9706 slave.cpp:948] Queuing task >> >> 'ct:1373041190990:0:add_latest_reservation_survey_events_partition' f >> >> or executor >> >> ct:1373041190990:0:add_latest_reservation_survey_events_partition of >> >> framework 'c >> >> hronos >> >> I0705 16:19:51.555027 9706 slave.cpp:511] Successfully attached file >> >> >> '/tmp/mesos/slaves/201307030043-2037266954-5050-15277-1517/frameworks/chronos/executors/ct:1373041190990:0:add_latest_reservation_survey_events_partition/runs/611ba128-557f-4b5e-8cf2-4d1ce60d618f' >> >> I0705 16:19:54.048754 9724 slave.cpp:2530] Current usage 42.18%. Max >> >> allowed age: 22.955009563956388hrs >> >> W0705 16:19:54.108963 9724 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:19:59.110787 9729 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:04.112406 9704 monitor.cpp:186] Failed to collect >> resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:09.114367 9705 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:14.116312 9706 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:19.118370 9699 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:24.120311 9701 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:29.122355 9700 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:34.123443 9722 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:39.125660 9718 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:44.127464 9724 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:49.129385 9725 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> I0705 16:20:51.555174 9703 slave.cpp:2482] Terminating executor >> >> ct:1373041190990:0:add_latest_reservation_survey_events_partition of >> >> framework chronos because it did not register within 1mins >> >> I0705 16:20:54.050434 9717 slave.cpp:2530] Current usage 42.18%. Max >> >> allowed age: 22.955009342481944hrs >> >> W0705 16:20:54.130730 9699 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:20:59.132472 9702 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:21:04.134557 9713 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> >> W0705 16:21:09.135619 9701 monitor.cpp:186] Failed to collect resource >> >> usage for executor 'executor_Task_Tracker_8023' of framework >> >> '201307030043-2037266954-5050-15277-0006': Future discarded >> > >> > >> > >> > >
