[ 
https://issues.apache.org/jira/browse/MESOS-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238353#comment-15238353
 ] 

Liqiang Lin commented on MESOS-5188:
------------------------------------

If that's the truth, the executor shall remove the stopped docker container in 
shutting down of executor, rather than try to stop the docker container.

> docker executor thinks task is failed when docker container was stopped
> -----------------------------------------------------------------------
>
>                 Key: MESOS-5188
>                 URL: https://issues.apache.org/jira/browse/MESOS-5188
>             Project: Mesos
>          Issue Type: Bug
>          Components: docker
>    Affects Versions: 0.28.0
>            Reporter: Liqiang Lin
>             Fix For: 0.29.0
>
>
> Test cases:
> 1. Launch a task with Swarm (on Mesos).
> {code}
> # docker -H 192.168.56.110:54375 run -d --cpu-shares 1 ubuntu sleep 300
> {code}
> 2. Then stop the docker container.
> {code}
> # docker -H 192.168.56.110:54375 ps
> CONTAINER ID        IMAGE               COMMAND             CREATED           
>   STATUS              PORTS               NAMES
> b4813ba3ed4d        ubuntu              "sleep 300"         9 seconds ago     
>   Up 8 seconds                            
> mesos1/mesos-2cd5576e-6260-4262-a62c-b0dc45c86c45-S1.1595e79b-aef2-44b6-a313-ad4ff8626958
> # docker -H 192.168.56.110:54375 stop b4813ba3ed4d
> b4813ba3ed4d
> {code}
> 3. Found the task is failed. See Mesos slave log,
> {code}
> I0407 09:10:57.606552 32307 slave.cpp:1508] Got assigned task 99ee7dc74861 
> for framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:10:57.608230 32307 slave.cpp:1627] Launching task 99ee7dc74861 for 
> framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:10:57.609979 32307 paths.cpp:528] Trying to chown 
> '/var/lib/mesos/slaves/2cd5576e-6260-4262-a62c-b0dc45c86c45-S0/frameworks/5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000/executors/99ee7dc74861/runs/250a169f-7aba-474d-a4f5-cd24ecf0e7d9'
>  to user 'root'
> I0407 09:10:57.615881 32307 slave.cpp:5586] Launching executor 99ee7dc74861 
> of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 with resources 
> cpus(*):0.1; mem(*):32 in work directory 
> '/var/lib/mesos/slaves/2cd5576e-6260-4262-a62c-b0dc45c86c45-S0/frameworks/5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000/executors/99ee7dc74861/runs/250a169f-7aba-474d-a4f5-cd24ecf0e7d9'
> I0407 09:12:18.458449 32307 slave.cpp:1845] Queuing task '99ee7dc74861' for 
> executor '99ee7dc74861' of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:12:18.459092 32307 slave.cpp:3711] No pings from master received 
> within 75secs
> I0407 09:12:18.460212 32307 slave.cpp:4593] Current disk usage 56.53%. Max 
> allowed age: 2.342613645432778days
> I0407 09:12:18.463484 32307 slave.cpp:928] Re-detecting master
> I0407 09:12:18.463969 32307 slave.cpp:975] Detecting new master
> I0407 09:12:18.464501 32307 slave.cpp:939] New master detected at 
> master@192.168.56.110:5050
> I0407 09:12:18.464848 32307 slave.cpp:964] No credentials provided. 
> Attempting to register without authentication
> I0407 09:12:18.465237 32307 slave.cpp:975] Detecting new master
> I0407 09:12:18.463611 32312 status_update_manager.cpp:174] Pausing sending 
> status updates
> I0407 09:12:18.465744 32312 status_update_manager.cpp:174] Pausing sending 
> status updates
> I0407 09:12:18.472323 32313 docker.cpp:1011] Starting container 
> '250a169f-7aba-474d-a4f5-cd24ecf0e7d9' for task '99ee7dc74861' (and executor 
> '99ee7dc74861') of framework '5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000'
> I0407 09:12:18.588739 32313 slave.cpp:1218] Re-registered with master 
> master@192.168.56.110:5050
> I0407 09:12:18.588927 32313 slave.cpp:1254] Forwarding total oversubscribed 
> resources
> I0407 09:12:18.589320 32313 slave.cpp:2395] Updating framework 
> 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 pid to 
> scheduler(1)@192.168.56.110:53375
> I0407 09:12:18.592079 32308 status_update_manager.cpp:181] Resuming sending 
> status updates
> I0407 09:12:18.592842 32313 slave.cpp:2534] Updated checkpointed resources 
> from  to
> I0407 09:12:18.592793 32308 status_update_manager.cpp:181] Resuming sending 
> status updates
> I0407 09:12:20.582041 32307 slave.cpp:2836] Got registration for executor 
> '99ee7dc74861' of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 from 
> executor(1)@192.168.56.110:40725
> I0407 09:12:20.584446 32307 docker.cpp:1308] Ignoring updating container 
> '250a169f-7aba-474d-a4f5-cd24ecf0e7d9' with resources passed to update is 
> identical to existing resources
> I0407 09:12:20.585093 32307 slave.cpp:2010] Sending queued task 
> '99ee7dc74861' to executor '99ee7dc74861' of framework 
> 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 at executor(1)@192.168.56.110:40725
> I0407 09:12:21.307077 32312 slave.cpp:3195] Handling status update 
> TASK_RUNNING (UUID: a7098650-cbf6-4445-8216-b5f658d2f5f4) for task 
> 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 from 
> executor(1)@192.168.56.110:40725
> I0407 09:12:21.308820 32308 status_update_manager.cpp:320] Received status 
> update TASK_RUNNING (UUID: a7098650-cbf6-4445-8216-b5f658d2f5f4) for task 
> 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:12:21.310058 32308 slave.cpp:3593] Forwarding the update 
> TASK_RUNNING (UUID: a7098650-cbf6-4445-8216-b5f658d2f5f4) for task 
> 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 to 
> master@192.168.56.110:5050
> I0407 09:12:21.310744 32308 slave.cpp:3503] Sending acknowledgement for 
> status update TASK_RUNNING (UUID: a7098650-cbf6-4445-8216-b5f658d2f5f4) for 
> task 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 to 
> executor(1)@192.168.56.110:40725
> I0407 09:12:21.315765 32307 status_update_manager.cpp:392] Received status 
> update acknowledgement (UUID: a7098650-cbf6-4445-8216-b5f658d2f5f4) for task 
> 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:16:13.943831 32306 slave.cpp:3195] Handling status update 
> TASK_FAILED (UUID: 552e07b9-f7b8-4442-8e5c-8e17b948f6dc) for task 
> 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 from 
> executor(1)@192.168.56.110:40725
> I0407 09:16:14.063880 32307 status_update_manager.cpp:320] Received status 
> update TASK_FAILED (UUID: 552e07b9-f7b8-4442-8e5c-8e17b948f6dc) for task 
> 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:16:14.064471 32312 slave.cpp:3593] Forwarding the update TASK_FAILED 
> (UUID: 552e07b9-f7b8-4442-8e5c-8e17b948f6dc) for task 99ee7dc74861 of 
> framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 to 
> master@192.168.56.110:5050
> I0407 09:16:14.065014 32312 slave.cpp:3503] Sending acknowledgement for 
> status update TASK_FAILED (UUID: 552e07b9-f7b8-4442-8e5c-8e17b948f6dc) for 
> task 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 to 
> executor(1)@192.168.56.110:40725
> I0407 09:16:14.071398 32307 status_update_manager.cpp:392] Received status 
> update acknowledgement (UUID: 552e07b9-f7b8-4442-8e5c-8e17b948f6dc) for task 
> 99ee7dc74861 of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:16:14.950019 32311 slave.cpp:3721] executor(1)@192.168.56.110:40725 
> exited
> I0407 09:16:14.993948 32308 docker.cpp:1997] Executor for container 
> '250a169f-7aba-474d-a4f5-cd24ecf0e7d9' has exited
> I0407 09:16:14.994104 32308 docker.cpp:1717] Destroying container 
> '250a169f-7aba-474d-a4f5-cd24ecf0e7d9'
> I0407 09:16:14.994875 32308 docker.cpp:1845] Running docker stop on container 
> '250a169f-7aba-474d-a4f5-cd24ecf0e7d9'
> I0407 09:16:14.996547 32308 slave.cpp:4097] Executor '99ee7dc74861' of 
> framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 exited with status 0
> I0407 09:16:14.996695 32308 slave.cpp:4201] Cleaning up executor 
> '99ee7dc74861' of framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000 at 
> executor(1)@192.168.56.110:40725
> I0407 09:16:14.997028 32312 gc.cpp:55] Scheduling 
> '/var/lib/mesos/slaves/2cd5576e-6260-4262-a62c-b0dc45c86c45-S0/frameworks/5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000/executors/99ee7dc74861/runs/250a169f-7aba-474d-a4f5-cd24ecf0e7d9'
>  for gc 6.99998846192days in the future
> I0407 09:16:14.997289 32308 slave.cpp:4289] Cleaning up framework 
> 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:16:14.997691 32307 status_update_manager.cpp:282] Closing status 
> update streams for framework 5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000
> I0407 09:16:14.997697 32312 gc.cpp:55] Scheduling 
> '/var/lib/mesos/slaves/2cd5576e-6260-4262-a62c-b0dc45c86c45-S0/frameworks/5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000/executors/99ee7dc74861'
>  for gc 6.99998845958222days in the future
> I0407 09:16:14.998423 32312 gc.cpp:55] Scheduling 
> '/var/lib/mesos/slaves/2cd5576e-6260-4262-a62c-b0dc45c86c45-S0/frameworks/5b84aad8-dd60-40b3-84c2-93be6b7aa81c-0000'
>  for gc 6.99998845141333days in the future
> I0407 09:16:18.470573 32308 slave.cpp:4593] Current disk usage 56.56%. Max 
> allowed age: 2.341048334320162days
> {code}
> Why docker executor thinks the task is failed when stop the docker container? 
> The docker container still exists. Should it be relay on state that docker 
> container is removed?
> Docker executor code
> {code}
>   void reaped(
>       ExecutorDriver* _driver,
>       const Future<Nothing>& run)
>   {
>     // Wait for docker->stop to finish, and best effort wait for the
>     // inspect future to complete with a timeout.
>     stop.onAny(defer(self(), [=](const Future<Nothing>&) {
>       inspect
>         .after(DOCKER_INSPECT_TIMEOUT, [=](const Future<Nothing>&) {
>           inspect.discard();
>           return inspect;
>         })
>         .onAny(defer(self(), [=](const Future<Nothing>&) {
>           CHECK_SOME(driver);
>           TaskState state;
>           string message;
>           if (!stop.isReady()) {
>             state = TASK_FAILED;
>             message = "Unable to stop docker container, error: " +
>                       (stop.isFailed() ? stop.failure() : "future discarded");
>           } else if (killed) {
>             state = TASK_KILLED;
>           } else if (!run.isReady()) {
>             state = TASK_FAILED;
>             message = "Docker container run error: " +
>                       (run.isFailed() ?
>                        run.failure() : "future discarded");
> {code}
> Another problem is, Mesos thinks the task is failed, and do some clean up for 
> executor. But the orphan docker container is not removed. I still can start 
> it manually, Mesos can not trace the resource usage of this orphan container.
> {code}
> # docker -H 192.168.56.110:54375 start b4813ba3ed4d
> b4813ba3ed4d
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to