Er, I could. At the moment it's pretty huge so maybe I'll just try and trim it down a bit. I've noticed that Chronos does the same, actually. There is a task that is 'active' and still holding onto resources yet it has already completed unsuccessfully with TASK_FAILED (16hrs ago!) state. Attached is a log of the events from the mesos slave that executed this particular Chronos task (before it continues to forward the same state over and over). Note that the last pair of lines is repeated ad-infinitum. I can confirm that this Chronos framework with the same ID is still running.
Sorry to switch frameworks suddenly - this was simpler because it was one task instead of 100s. Jim On 24 November 2015 at 17:57, Vinod Kone <[email protected]> wrote: > Can you paste the logs? > > On Tue, Nov 24, 2015 at 2:16 AM, James Vanns <[email protected]> wrote: > >> Hi again list. >> >> Mesos 0.24 >> C++ Framework (still using the Protobufs based comms, not REST) >> >> My framework appears to be holding onto offers (somehow) from tasks that >> are finished!? I don't understand why. The task comprises of a shell >> command that executes within a docker container. >> The return code to the OS from the shell command is indeed zero for >> success, which Mesos honours and transitions to TASK_FINISHED state. >> However, using the UI these still register as 'active' (though acknowledged >> as FINISHED) and thus the resources are not yet freed. >> >> Any pointers appreciated! >> >> Cheers, >> >> Jim >> >> -- >> Senior Code Pig >> Industrial Light & Magic >> > > -- -- Senior Code Pig Industrial Light & Magic
I1124 16:57:10.782155 104 slave.cpp:1739] Sending queued task 'ct:1448384217181:2:olio:' to executor 'ct:1448384217181:2:olio:' of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:11.490393 107 slave.cpp:2696] Handling status update TASK_RUNNING (UUID: a3dd3fc7-c5cf-4d7a-bbf9-efe8e799b54a) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 from executor(1)@172.20.121.112:43306 I1124 16:57:11.490633 105 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: a3dd3fc7-c5cf-4d7a-bbf9-efe8e799b54a) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:11.491034 105 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: a3dd3fc7-c5cf-4d7a-bbf9-efe8e799b54a) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:11.491111 107 slave.cpp:2696] Handling status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 from executor(1)@172.20.121.112:43306 I1124 16:57:11.493294 104 slave.cpp:2975] Forwarding the update TASK_RUNNING (UUID: a3dd3fc7-c5cf-4d7a-bbf9-efe8e799b54a) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to [email protected]:5050 I1124 16:57:11.493482 104 slave.cpp:2905] Sending acknowledgement for status update TASK_RUNNING (UUID: a3dd3fc7-c5cf-4d7a-bbf9-efe8e799b54a) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to executor(1)@172.20.121.112:43306 I1124 16:57:11.499173 105 status_update_manager.cpp:394] Received status update acknowledgement (UUID: a3dd3fc7-c5cf-4d7a-bbf9-efe8e799b54a) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:11.499286 105 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: a3dd3fc7-c5cf-4d7a-bbf9-efe8e799b54a) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:11.561130 107 status_update_manager.cpp:322] Received status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:11.561177 107 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:11.562505 109 slave.cpp:2975] Forwarding the update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to [email protected]:5050 I1124 16:57:11.562604 109 slave.cpp:2905] Sending acknowledgement for status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to executor(1)@172.20.121.112:43306 I1124 16:57:12.566434 107 docker.cpp:1584] Executor for container 'db9f7671-13cb-430f-b74f-3fc1df898f89' has exited I1124 16:57:12.566491 107 docker.cpp:1382] Destroying container 'db9f7671-13cb-430f-b74f-3fc1df898f89' I1124 16:57:12.566517 107 docker.cpp:1486] Running docker stop on container 'db9f7671-13cb-430f-b74f-3fc1df898f89' I1124 16:57:12.566627 103 slave.cpp:3399] Executor 'ct:1448384217181:2:olio:' of framework 20151119-165710-4000912556-5050-1-0052 exited with status 0 W1124 16:57:21.563534 108 status_update_manager.cpp:477] Resending status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:21.563688 108 slave.cpp:2975] Forwarding the update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to [email protected]:5050 I1124 16:57:26.051758 109 http.cpp:174] HTTP GET for /slave(1)/state.json from 172.20.121.148:48777 with User-Agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0' W1124 16:57:41.564505 105 status_update_manager.cpp:477] Resending status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:57:41.564697 105 slave.cpp:2975] Forwarding the update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to [email protected]:5050 I1124 16:58:01.992205 108 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127850156366days W1124 16:58:21.565486 105 status_update_manager.cpp:477] Resending status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:58:21.565670 107 slave.cpp:2975] Forwarding the update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to [email protected]:5050 I1124 16:59:01.992918 103 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127487829167days W1124 16:59:41.566162 103 status_update_manager.cpp:477] Resending status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 16:59:41.566342 103 slave.cpp:2975] Forwarding the update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to [email protected]:5050 I1124 17:00:01.993217 108 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948126400847547days I1124 17:01:01.994376 104 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127125501956days I1124 17:02:01.995360 109 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127125501956days W1124 17:02:21.566828 104 status_update_manager.cpp:477] Resending status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 17:02:21.567008 104 slave.cpp:2975] Forwarding the update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to [email protected]:5050 I1124 17:03:01.996482 109 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127125501956days I1124 17:04:01.997622 109 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127125501956days I1124 17:05:01.997980 110 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127125501956days I1124 17:06:01.998822 103 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127125501956days I1124 17:07:01.999371 106 slave.cpp:3885] Current disk usage 19.31%. Max allowed age: 4.948127125501956days W1124 17:07:41.567502 107 status_update_manager.cpp:477] Resending status update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 I1124 17:07:41.567682 107 slave.cpp:2975] Forwarding the update TASK_FAILED (UUID: ab530800-346a-4ba5-9455-7d35c90adf19) for task ct:1448384217181:2:olio: of framework 20151119-165710-4000912556-5050-1-0052 to [email protected]:5050
